Research map / Video / Video segmentation and tracking
S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S2D, an unsupervised video instance segmentation method that trains on real videos by discovering sparse temporally-coherent keymasks from CutS3D single-frame masks and densifying them via sparse-to-dense distillation with a Temporal DropLoss.
Problem
Unsupervised video instance segmentation has relied on synthetic videos made by shifting ImageNet object masks, which fail to model realistic instance motion; the paper asks whether real videos can train such models without human labels.
Approach
Filter noisy single-frame unsupervised masks into sparse keymasks using point-track visibility grouping and point-mask Jaccard matching, then train a VideoMask2Former with a Temporal DropLoss (loss applied only to annotated frames) and an EMA teacher to implicitly propagate masks to dense predictions, followed by a second self-distillation round.
Key results
- In-domain, S2D reaches 26.2 AP on YouTube-VIS 2019 versus 24.5 for VideoCutLER, and 20.1 AP on YouTube-VIS 2021 versus 18.0.
- Zero-shot with 13K real videos, S2D reaches 51.4 AP50 on YouTube-VIS 2019 versus 48.2 for VideoCutLER trained on 1.3M synthetic videos.
- On DAVIS-all, S2D reaches 50.6 J&F versus 44.9 for VideoCutLER.
Surprising findings
- Single-frame masks trained with an image Mask2Former alone already perform competitively with VideoCutLER on YTVIS 2021 (36.5 AP50 mask).
- Using 13K real videos outperforms a model trained on 1.3M synthetic videos in zero-shot evaluation.
Limitations
- Struggles with small objects.
- Zero-shot model performs poorly on driving-domain data such as Cityscapes.
- Relies on deep motion cues (point tracks) for temporal coherence rather than deep temporal features.
Experimental setup
Datasets: YouTube-VIS 2019 YouTube-VIS 2021 DAVIS-all VIPSeg MOSEv1 SA-V ImageNet
Benchmarks: YouTube-VIS 2019 YouTube-VIS 2021 YouTube-VIS 2022 DAVIS-all OVIS Cityscapes
Metrics: AP AP50 J&F J-Mean F-Mean
Baselines: VideoCutLER CutLER OCLR MotionGroup DINO propagation DINO
Builds on: CutS3D VideoMask2Former VideoCutLER Mask2Former CutLER
Closest papers by meaning
- Mask-Free Video Instance Segmentation (CVPR 2023 · similarity 0.89)
- VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation (CVPR 2024 · similarity 0.88)
- Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation (CVPR 2025 · similarity 0.85)
- Robust and Consistent Online Video Instance Segmentation via Instance Mask Propagation (AAAI 2025 · similarity 0.85)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.85)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.