Research map / Video / Video segmentation and tracking
VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoCutLER, an unsupervised video instance segmentation method that trains a VideoMask2Former on synthetic videos built from ImageNet images and MaskCut pseudo-masks, without optical flow or natural videos.
Problem
Unsupervised video instance segmentation typically relies on optical flow motion cues, which fail for small, static, or overlapping objects; the paper asks whether video instance segmentation can be learned without motion signals or natural videos.
Approach
Generate pseudo-masks for multiple objects per image with MaskCut, then synthesize training videos by pasting augmented masked objects from a second image onto a duplicated first image (ImageCut2Video), and train a video instance segmentation model on these mask trajectories so it tracks by appearance similarity.
Key results
- Achieves 50.7% AP50 (class-agnostic) on YouTubeVIS-2019 zero-shot, versus 4.8% for prior SOTA OCLR.
- As a pretrained model for supervised VIS, exceeds DINO by 15.9% AP on YouTubeVIS-2019 (38.9 vs 23.0).
- On DAVIS2017 J&F, reaches 43.6 vs 39.6 for OCLR (w/ flow) without optical flow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train VideoCutLER rather than OCLR | Unsupervised video instance segmentation · YouTubeVIS-2019 train split | AP50: 4.8 -> 50.7 | Helped |
| Train VideoCutLER rather than CutLER | Unsupervised video instance segmentation · YouTubeVIS-2019 train split | AP50: 37.5 -> 50.7 | Helped |
| Train VideoCutLER rather than OCLR | Unsupervised video instance segmentation · YouTubeVIS-2021 train split | AP50: 4.4 -> 38.9 | Helped |
| Train VideoCutLER rather than OCLR | Unsupervised video instance segmentation · DAVIS2017 val | J &F: 39.6 -> 43.6 | Helped |
| Train VideoCutLER rather than OCLR | Unsupervised video instance segmentation · DAVIS2017-Motion val | J &F: 55.1 -> 57.3 | Helped |
| Pretrain with VideoCutLER instead of DINO before supervised fine-tuning | VideoMask2Former · YouTubeVIS-2019 val | AP: 23.0 -> 38.9 | Helped |
| Use frame size 180 instead of 360 (ablation) | VideoCutLER · YouTubeVIS-2019 | AP50: 50.7 -> 49.9 | Hurt |
| Use frame size 480 instead of 360 (ablation) | VideoCutLER · YouTubeVIS-2019 | AP50: 50.7 -> 50.4 | Hurt |
| Train with 2 frames instead of 3 (ablation) | VideoCutLER · YouTubeVIS-2019 | AP50: 50.7 -> 49.8 | Hurt |
| Train with 4 frames instead of 3 (ablation) | VideoCutLER · YouTubeVIS-2019 | AP50: 50.7 -> 50.4 | Hurt |
| Add ImageCut2Video augmentations (ablation) | VideoCutLER · YouTubeVIS-2019 | AP50: 47.8 -> 50.7 | Helped |
| Add brightness augmentation only (ablation) | VideoCutLER · YouTubeVIS-2019 | AP50: 47.8 -> 48.1 | Helped |
Surprising findings
- Synthetic videos built by pasting masks from two ImageNet images are sufficient to train a video instance segmentation model that generalizes to real YouTube videos without any video data or flow.
Limitations
- Synthetic training videos are temporally simple (two or three frames), and the authors suggest natural videos could further improve results.
- Evaluation on DAVIS is unfavorable to the method since DAVIS annotates only prominent moving objects.
- Performance gap to supervised learning remains (AP50 gap of 11.0 points on YouTubeVIS-2019).
Experimental setup
Datasets: ImageNet-1K YouTubeVIS-2019 YouTubeVIS-2021 DAVIS2017 DAVIS2017-Motion
Benchmarks: YouTubeVIS-2019 YouTubeVIS-2021 DAVIS2017 DAVIS2017-Motion
Metrics: AP AP50 AP75 APS APM APL AR10 AR100 J&F J
Baselines: OCLR MotionGroup CutLER DINO MaskTrack R-CNN CRW
Builds on: CutLER MaskCut VideoMask2Former DINO Normalized Cuts
Closest papers by meaning
- InstanceFormer: An Online Video Instance Segmentation Framework (AAAI 2023 · similarity 0.89)
- Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (CVPR 2020 · similarity 0.89)
- Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation (CVPR 2025 · similarity 0.88)
- Mask-Free Video Instance Segmentation (CVPR 2023 · similarity 0.88)
- S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation (NeurIPS 2026 · similarity 0.88)
- End-to-End Video Instance Segmentation With Transformers (CVPR 2021 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.