Research map / Video / Video segmentation and tracking
SyncVIS: Synchronized Video Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SyncVIS, a DETR-based video instance segmentation framework that synchronously updates video-level and frame-level query embeddings in each decoder layer and optimizes bipartite matching over short sub-clips, reaching SOTA on YouTube-VIS 2019/2021/2022 and OVIS.
Problem
Query-based video instance segmentation methods use asynchronous or cascade designs between frame-level and video-level queries, which degrade as the number of input frames grows and make video-level bipartite matching hard to optimize.
Approach
Bidirectionally and synchronously exchange information between frame-level and video-level embeddings in every decoder layer (with top-N_k key/value selection and momentum update lambda), and split long training videos into sub-clips for separate video-level bipartite matching losses.
Key results
- SyncVIS (online, Swin-L) reaches 67.1 AP on YouTube-VIS 2019, 1.5 points above CTVIS (65.6).
- SyncVIS (offline, ResNet-50) reaches 54.2 AP on YouTube-VIS 2019, above DVIS (52.6) and VITA (49.8).
- SyncVIS reaches 36.3 AP on OVIS with ResNet-50 and 50.8 AP with Swin-L, outperforming DVIS by 2.2 and 0.9 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add synchronized video-frame modeling (ablation) | Mask2Former-VIS · YouTube-VIS 2019 | AP: +5.2 points | Helped |
| add synchronized embedding optimization (ablation) | Mask2Former-VIS · YouTube-VIS 2019 | AP: 45.1 -> 46.7 | Helped |
| add synchronized embedding optimization (ablation) | Mask2Former-VIS · YouTube-VIS 2021 | AP: 39.8 -> 41.3 | Helped |
| add synchronized embedding optimization (ablation) | Mask2Former-VIS · OVIS | AP: 10.6 -> 12.3 | Helped |
| replace class-score aggregation selection with query-similarity selection (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 49.7 | Hurt |
| replace class-score aggregation selection with mask-similarity selection (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 48.2 | Hurt |
| set aggregation embedding count Nk to 5 instead of 10 (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 51.1 | Hurt |
| set aggregation embedding count Nk to 25 instead of 10 (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 50.9 | Hurt |
| set aggregation embedding count Nk to 50 instead of 10 (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 49.3 | Hurt |
| set aggregation embedding count Nk to 100 instead of 10 (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 47.5 | Hurt |
| set sub-clip size Ts to 1 instead of 3 (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 50.9 | Hurt |
| set sub-clip size Ts to 2 instead of 3 (ablation) | SyncVIS · YouTube-VIS 2019 | AP: 51.5 -> 51.3 | Hurt |
Surprising findings
- Mask2Former-VIS performance degrades as more input frames are used during training (about 1.5 AP drop at ten frames), which the authors call counter-intuitive.
- Mask similarity as an embedding-selection criterion performs worse than query similarity or class prediction.
Limitations
- Struggles with very crowded or heavily occluded scenes, as acknowledged by the authors.
- Sub-clip size T_s must be tuned per dataset (3 for YouTube-VIS 2019/2021, 2 for YouTube-VIS 2022 and OVIS).
Experimental setup
Datasets: COCO YouTube-VIS 2019 YouTube-VIS 2021 YouTube-VIS 2022 OVIS
Benchmarks: YouTube-VIS 2019 YouTube-VIS 2021 YouTube-VIS 2022 OVIS
Metrics: AP AP50 AP75 AR1 AR10
Baselines: Mask2Former-VIS VITA DVIS CTVIS IDOL MinVIS SeqFormer TCOVIS IFC TeViT
Builds on: Mask2Former Mask2Former-VIS CTVIS VITA IDOL
Closest papers by meaning
- Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation (CVPR 2025 · similarity 0.88)
- Robust and Consistent Online Video Instance Segmentation via Instance Mask Propagation (AAAI 2025 · similarity 0.86)
- Exploiting Temporal State Space Sharing for Video Semantic Segmentation (CVPR 2025 · similarity 0.86)
- SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis (CVPR 2025 · similarity 0.86)
- Decoupled Motion Expression Video Segmentation (CVPR 2025 · similarity 0.86)
- Anchor-Aware Similarity Cohesion in Target Frames Enables Predicting Temporal Moment Boundaries in 2D (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.