Research map / Detection and segmentation / CNN features
Putting the Object Back into Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cutie is a video object segmentation network that performs top-down object-level memory reading with a query-based object transformer and foreground-background masked attention, improving robustness on MOSE.
Problem
Semi-supervised video object segmentation with pixel-level memory matching is noisy under distractors and occlusion, causing large performance drops on challenging benchmarks like MOSE.
Approach
Replace bottom-up pixel-only memory readout with an object transformer that lets a small set of object queries (split into foreground and background halves via masked attention) iteratively interact with pixel features, plus a compact object memory built by mask pooling, while avoiding spatial-spatial attention for efficiency.
Key results
- On MOSE validation, Cutie-base trained with MOSE reaches 68.3 J&F vs 59.6 for XMem, an improvement of about 8.7 points.
- Cutie-small runs at 45.5 FPS on YouTubeVOS on a V100, versus 11.7 FPS for DeAOT-R50 and 22.6 FPS for XMem.
- Ablation: removing foreground-background masked attention drops MOSE J&F from 67.3 to 63.8, and bottom-up-only reading drops it to 65.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Cutie-base instead of DEVA | video object segmentation · MOSE val (trained without MOSE) | J &F: 60.0 -> 64.0 | Helped |
| use Cutie-base instead of DEVA | video object segmentation · DAVIS-17 val | J &F: 86.8 -> 88.8 | Helped |
| use Cutie-base instead of DEVA | video object segmentation · DAVIS-17 test | J &F: 82.3 -> 84.2 | Helped |
| use Cutie-base instead of DeAOT-R50 | video object segmentation · YouTubeVOS-2019 val | G: 85.6 -> 86.1 | Helped |
| use Cutie-base instead of DEVA | video object segmentation · MOSE val (trained with MOSE) | J &F: 66.0 -> 68.3 | Helped |
| use Cutie-base with long-term memory instead of DeAOT with unbounded memory | video object segmentation · BURST val | All: 56.4 -> 58.4 | Helped |
| add an object transformer (ablation) | Cutie-small · MOSE val | J &F: 65.2 -> 67.3±0.36 | Helped |
| add pixel memory to the top-down-only model (ablation) | Cutie-small · MOSE val | J &F: 40.7±1.62 -> 67.3±0.36 | Helped |
| add object memory (ablation) | Cutie-small · MOSE val | J &F: 66.9±0.26 -> 67.3±0.36 | Helped |
| add static object queries (ablation) | Cutie-small · MOSE val | J &F: 67.2±0.10 -> 67.3±0.36 | No clear effect |
| use foreground-background masked attention instead of foreground-only masked attention (ablation) | Cutie-small · MOSE val | J &F: 66.7±0.21 -> 67.3±0.36 | Helped |
| add masked attention (ablation) | Cutie-small · MOSE val | J &F: 63.8±1.06 -> 67.3±0.36 | Helped |
Surprising findings
- The object queries are reported as not useful once the object memory is present (removing them gives 67.2 vs 67.3 J&F).
- Cutie is insensitive to the number of object queries (N=8, 16, 32 give similar results).
Limitations
- Fails when highly similar objects move in close proximity or occlude each other.
- Object memory and queries cannot pick up sufficiently discriminative features in such cases.
Experimental setup
Datasets: DAVIS YouTubeVOS MOSE BURST OVIS static image datasets
Benchmarks: MOSE DAVIS 2017 YouTubeVOS-2019 BURST LVOS
Metrics: J&F J F HOTA FPS GPU memory
Baselines: XMem DeAOT STCN AOT RDE DEVA SimVOS JointFormer ISVOS
Builds on: XMem Mask2Former DeAOT DEVA
Closest papers by meaning
- Breaking the "Object" in Video Object Segmentation (CVPR 2023 · similarity 0.88)
- HODOR: High-Level Object Descriptors for Object Re-Segmentation in Video Learned From Static Images (CVPR 2022 · similarity 0.88)
- Look Before You Match: Instance Understanding Matters in Video Object Segmentation (CVPR 2023 · similarity 0.87)
- Deep Extreme Cut: From Extreme Points to Object Segmentation (CVPR 2018 · similarity 0.87)
- RMem: Restricted Memory Banks Improve Video Object Segmentation (CVPR 2024 · similarity 0.86)
- LiVOS: Light Video Object Segmentation with Gated Linear Matching (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.