Research map / Detection and segmentation / CNN features
Robust and Consistent Online Video Instance Segmentation via Instance Mask Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoCoVIS, an end-to-end online video instance segmentation model that propagates object queries through mask predictions (instance mask propagation) with a modified label assignment, and advocates Tube-Boundary AP and APPool metrics for consistency.
Problem
Tracking-by-detection online VIS methods produce temporally inconsistent masks at pixel and instance levels, and standard AP metric fails to capture boundary quality and cross-category calibration.
Approach
Replace GenVIS's object decoder with direct query decoding onto pixel features, and calibrate current-frame pixel embeddings using object queries spatially mapped by previous-frame mask predictions (instance mask propagation), paired with UVLA++ label assignment that handles disappeared objects.
Key results
- With Swin-L on OVIS, RoCoVIS reaches 48.5 AP versus 47.1 for DVIS-online and 46.9 for CTVIS.
- On HQ-YTVIS test with ResNet-50, RoCoVIS achieves 39.6 Tube-Boundary AP versus 37.5 for GenVIS.
- Ablation on HQ-YTVIS val: APPool rises from 50.1 (GenVIS) to 54.8 with all components.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the object decoder with direct query decoding (ablation) | GenVIS · HQ-YTVIS val | AP: 51.4 -> 53.5 | Helped |
| add instance mask propagation (ablation) | Direct Query Decoding · HQ-YTVIS val | AP: 53.5 -> 52.3 | Hurt |
| add instance mask propagation (ablation) | Direct Query Decoding · HQ-YTVIS val | APB: 37.0 -> 37.5 | Helped |
| add UVLA++ label assignment (ablation) | Direct Query Decoding + Instance Mask Prop. · HQ-YTVIS val | AP: 52.3 -> 54.1 | Helped |
| add UVLA++ label assignment (ablation) | Direct Query Decoding + Instance Mask Prop. · HQ-YTVIS val | APB: 37.5 -> 39.0 | Helped |
| add UVLA++ label assignment (ablation) | Direct Query Decoding + Instance Mask Prop. · HQ-YTVIS val | APPool: 52.7 -> 54.8 | Helped |
| use RoCoVIS instead of DVISonline | Swin-L online VIS · OVIS val | AP: 47.1 -> 48.5 | Helped |
| use RoCoVIS instead of GenVIS | ResNet-50 online VIS · HQ-YTVIS test | AP: 55.2 -> 57.2 | Helped |
| use RoCoVIS instead of GenVIS | ResNet-50 online VIS · VIPSeg-things val | AP: 24.2 -> 24.9 | Helped |
Surprising findings
- Instance mask propagation alone slightly decreased standard AP on HQ-YTVIS despite improving boundary AP, which the authors attribute to UVLA error propagation for temporarily disappeared objects.
Limitations
- End-to-end learning incurs higher training cost than decoupled tracking-by-detection methods.
- Integrating a large number of historical frames remains an open challenge.
- Standard YouTube-VIS and OVIS validation lacks public ground truth, limiting new-metric evaluation to HQ-YTVIS and VIPSeg.
Experimental setup
Datasets: YouTube-VIS 2019 YouTube-VIS 2022 OVIS HQ-YTVIS VIPSeg
Benchmarks: YouTube-VIS 2022 OVIS HQ-YTVIS VIPSeg-things
Metrics: AP AP50 AP75 AR1 AR10 Tube-Boundary AP APPool
Baselines: MinVIS GenVIS IDOL CTVIS TCOVIS DVISonline SeqFormer VMT
Builds on: GenVIS Mask2Former UVLA APPool metric (Dave et al. 2021) VMT
Closest papers by meaning
- Mask-Free Video Instance Segmentation (CVPR 2023 · similarity 0.90)
- Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (CVPR 2020 · similarity 0.90)
- InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation (NeurIPS 2022 · similarity 0.89)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.87)
- Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation (CVPR 2025 · similarity 0.87)
- SyncVIS: Synchronized Video Instance Segmentation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.