Research map / Detection and segmentation / Open-set and few-shot detection
Multi-Granularity Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MUG-VOS, a large automatically collected multi-granularity video object segmentation dataset (SAM plus optical flow and IoU tracking) with a human-verified test set, and a memory-based SAM propagation model (MMPM) that outperforms XMem and DEVA on it.
Problem
Existing video segmentation and VOS benchmarks annotate only salient foreground objects, so models trained on them fail to segment non-salient objects, parts, stuff, and backgrounds in real-world videos.
Approach
Automatically generate dense multi-granularity mask tracks by prompting SAM with grid points on the first frame and propagating masks with optical-flow-warped points and IoU-based candidate selection, then train a SAM-encoder memory model (MMPM) with temporal and sequential memory on this data.
Key results
- MMPM reaches 86.1 J&F on MUG-VOS test, versus 85.6 for DEVA and 83.0 for XMem, and 78.3 for SAM-PT.
- On DAVIS-2017 validation (trained only on MUG-VOS train), MMPM achieves 69.1 J&F versus 64.1 for DEVA and 63.0 for XMem.
- Memory ablation: no memory gives 81.8 J&F, while both sequential and temporal memory gives 86.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MMPM instead of the strongest listed baseline, DEVA | Video segmentation on DAVIS-2017 validation · DAVIS-2017 validation | J&F: 85.6 -> 86.1 | Helped |
| use MMPM instead of the strongest listed baseline, DEVA | Video segmentation on DAVIS-2017 validation · DAVIS-2017 validation | J&F: 64.1 -> 69.1 | Helped |
| add FIFO filtering instead of retaining the first inserted values (ablation) | MMPM memory · MUG-VOS Test | J&F: 85.0 -> 85.9 | Helped |
| add random filtering instead of FIFO filtering (ablation) | MMPM memory · MUG-VOS Test | J&F: 85.9 -> 86.0 | Helped |
| preserve the first and last entries during random filtering (ablation) | MMPM memory · MUG-VOS Test | J&F: 86.0 -> 86.1 | Helped |
| add sequential memory to a model without memory (ablation) | MMPM · MUG-VOS Test | J&F: 81.8 -> 81.9 | Helped |
| add temporal memory to a model without memory (ablation) | MMPM · MUG-VOS Test | J&F: 81.8 -> 85.7 | Helped |
| add sequential memory to temporal memory (ablation) | MMPM · MUG-VOS Test | J&F: 85.7 -> 86.1 | Helped |
| increase memory update interval from r=1 to r=5 (ablation) | MMPM · MUG-VOS Test | J&F: 66.6 -> 86.1 | Helped |
| increase temporal memory values from N=5 to N=10 (ablation) | MMPM · MUG-VOS Test | J&F: 85.7 -> 86.1 | Helped |
| increase temporal memory values from N=10 to N=15 (ablation) | MMPM · MUG-VOS Test | J&F: 86.1 -> 86.1 | No clear effect |
Surprising findings
- XMem and DEVA, strong on standard VOS benchmarks, perform worse on MUG-VOS than the proposed model; the authors argue MUG-VOS is harder than existing VOS benchmarks.
- Keeping only the first and last temporal memory values on random filtering was best, though gains over plain random filtering are small (86.0 to 86.1).
Limitations
- Training masks are automatically generated by SAM and flow with no human verification; only the test set has quality assurance.
- The test set is small (30 videos from DAVIS-17 validation).
- The SAM-based IoU tracking baseline fails under large motion and occlusion.
Experimental setup
Datasets: HD-VILA-100M DAVIS-2017 SA-1B YouTube-VOS UVO
Benchmarks: MUG-VOS test DAVIS-2017 validation
Metrics: J&F J F
Baselines: XMem DEVA SAM-PT PerSAM PerSAM-F SAM (Grid 32) + IoU SAM (Grid 64) + IoU
Builds on: Segment Anything XMem DEVA STCN
Closest papers by meaning
- MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation (CVPR 2019 · similarity 0.88)
- Point-VOS: Pointing Up Video Object Segmentation (CVPR 2024 · similarity 0.88)
- Learning Video Object Segmentation From Unlabeled Videos (CVPR 2020 · similarity 0.87)
- M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (CVPR 2025 · similarity 0.87)
- Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation (AAAI 2024 · similarity 0.86)
- RMem: Restricted Memory Banks Improve Video Object Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.