Research map / Detection and segmentation / Weakly supervised detection and segmentation
Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MinMaxVIS, which trains video instance segmentation from ~1-10% labeled target-domain images plus retrieved high-confidence pseudo-labeled SA-1B images, using selective gradient backpropagation and an auxiliary instance-association decoder on augmented image pairs.
Problem
Video instance segmentation normally requires costly per-frame video annotations with instance association labels; the paper asks whether VIS can be trained with only a small labeled target-domain image set plus large unlabeled general-domain images.
Approach
Use a preliminary Mask2Former model trained on a few labeled target images to retrieve high-confidence pseudo-labeled instances from a huge unlabeled pool (SA-1B), then train an image-based VIS model with selective gradient backpropagation to handle pseudo-label false negatives and an auxiliary decoder with an instance-association loss on augmented image pairs that simulate video continuity.
Key results
- MinMaxVIS with Swin-L reaches 62.2 mAP on YouTube-VIS 2019 with 2% labeled data, versus 62.2 vs 61.6 for MinVIS trained on 100% data (0.6 mAP higher than MinVIS full-set)
- With ResNet-50 on YouTube-VIS 2019 (2%), MinMaxVIS gets 50.0 mAP vs MinVIS at 37.8 with the same low-data setting
- Ablation on YouTube-VIS 2019 (2%, Swin-L): MinVIS baseline 58.5 mAP rises to 60.3 with high-precision retrieval, 61.4 with selective gradient backpropagation, and 62.2 with instance association
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add high-precision retrieval and train with retrieved pseudo-labeled images (ablation) | MinVIS baseline · YouTube-VIS 2019 (2%) | mAP: 58.5% to 60.3% | Helped |
| Add selective gradient backpropagation (ablation) | MinMaxVIS with high-precision retrieval · YouTube-VIS 2019 (2%) | mAP: 60.3% to 61.4% | Helped |
| Add instance association (ablation) | MinMaxVIS with selective gradient backpropagation · YouTube-VIS 2019 (2%) | mAP: 61.4% to 62.2% | Helped |
| Use truncation-weight SGB instead of confidence-weight SGB (ablation) | MinMaxVIS · YouTube-VIS 2019 (2%) | mAP: 60.7 -> 62.2 | Helped |
| Use truncation-weight SGB instead of squared-confidence-weight SGB (ablation) | MinMaxVIS · YouTube-VIS 2019 (2%) | mAP: 61.7 -> 62.2 | Helped |
| Set truncation threshold β to 0.0 rather than β=0.5 (ablation) | MinMaxVIS SGB · YouTube-VIS 2019 (2%) | mAP: 62.2 -> 61.4 | Hurt |
| Set truncation threshold β to 0.3 rather than β=0.5 (ablation) | MinMaxVIS SGB · YouTube-VIS 2019 (2%) | mAP: 62.2 -> 61.2 | Hurt |
| Set truncation threshold β to 0.7 rather than β=0.5 (ablation) | MinMaxVIS SGB · YouTube-VIS 2019 (2%) | mAP: 62.2 -> 61.6 | Hurt |
| Set truncation threshold β to 0.9 rather than β=0.5 (ablation) | MinMaxVIS SGB · YouTube-VIS 2019 (2%) | mAP: 62.2 -> 60.1 | Hurt |
| Also push background-background similarities in instance association (ablation) | MinMaxVIS instance association · YouTube-VIS 2019 (2%) | mAP: 62.2 -> 60.2 | Hurt |
| Use layer 6 instead of layer 3 features for instance association (ablation) | MinMaxVIS · YouTube-VIS 2019 (2%) | mAP: 62.2 -> 61.5 | Hurt |
| Use layer 9 instead of layer 3 features for instance association (ablation) | MinMaxVIS · YouTube-VIS 2019 (2%) | mAP: 62.2 -> 61.4 | Hurt |
Surprising findings
- Adding background-background push to the instance association loss hurts performance (60.2 vs 62.2)
- Deeper-layer auxiliary inputs perform worse; shallower decoder layer 3 gives best instance association
Limitations
- Relies on a high retrieval confidence threshold (0.99, lowered to 0.9 for sparse categories), so recall of pseudo-labels may be limited
- Baselines adapted from video to image-based training by simulated augmentations, which may not reflect their original performance
- Evaluation limited to three VIS benchmarks with mAP only
Experimental setup
Datasets: YouTube-VIS 2019 YouTube-VIS 2021 OVIS SA-1B
Benchmarks: YouTube-VIS 2019 YouTube-VIS 2021 OVIS
Metrics: mAP
Baselines: MinVIS VITA DVIS DVIS++ GenVIS DVIS-DAQ
Builds on: MinVIS Mask2Former Hungarian matching
Closest papers by meaning
- MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training (NeurIPS 2022 · similarity 0.94)
- A Generalized Framework for Video Instance Segmentation (CVPR 2023 · similarity 0.91)
- VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation (CVPR 2024 · similarity 0.88)
- SyncVIS: Synchronized Video Instance Segmentation (NeurIPS 2024 · similarity 0.88)
- WISH: Weakly Supervised Instance Segmentation using Heterogeneous Labels (CVPR 2025 · similarity 0.87)
- Aligning Instance Brownian Bridge with Texts for Open-Vocabulary Video Instance Segmentation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.