Robust online video instance segmentation with track queries

With support of data and maturity of model, deep learning has been successful in high level understanding on videos. Recently, transformer-based methods have achieved impressive results on Video Instance Segmentation (VIS). However, most of these top-performing methods run in an offline manner by processing the entire video clip at once to predict instance mask volumes. This makes them incapable of handling the long videos that appear in challenging new video instance segmentation datasets like Unidentified Video Objects (UVO) and Occluded Video Instance Segmentation (OVIS). We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark and considerably outperforms them on UVO and OVIS. This method, called Robust Online Video Segmentation (ROVIS), augments the Mask2Former image instance segmentation model with track queries, a lightweight mechanism for carrying track information from frame to frame, originally introduced by the TrackFormer method for multi-object tracking. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos. Lastly, we extend ROVIS to the Refer-VOS task which is highly relevant to VIS, such that ROVIS can make segmentation predictions based on language input, and present our preliminary results.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Zhan, Zitong

Contributors dc:contributor

• Lazebnik, Svetlana

• Computer Vision
• Video Tracking

Statement dc:rights

• Copyright 2023 Zitong Zhan

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/121199

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F121199