Research map / Detection and segmentation / Open-set and few-shot detection
Guided Slot Attention for Unsupervised Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GSA-Net, a guided slot attention network for unsupervised video object segmentation that initializes slots from encoder features, selects features via KNN filtering, and refines slots with a feature aggregation transformer.
Problem
Unsupervised video object segmentation must find the most salient object amid complex backgrounds and multiple similar foreground objects, and existing slot attention methods with randomly initialized slots degrade in real-world scenes.
Approach
Replace randomly initialized slots with guided foreground/background slots pooled from target-frame features, and iteratively refine them with KNN-filtered features aggregated from local target-frame and global reference-frame features via a feature aggregation transformer.
Key results
- Achieves 87.7 GM on DAVIS-16 with ResNet101 backbone (88.2 with MiT-b2) and 79.2 / 82.3 JM on FBMS, state-of-the-art among compared methods.
- Guided slots improve DAVIS-16 GM from 83.7 to 84.1 (baseline vs guided) and to 86.1 when combined with KNN filtering in the ablation table.
- With test time augmentation, the MiT-b2 variant reaches 88.9 GM on DAVIS-16, versus HFAN* at 88.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use guided slots instead of randomly initialized slots (ablation) | GSA-Net · DAVIS-16 | GM: 83.7 -> 84.1 | Helped |
| Use guided slots instead of randomly initialized slots (ablation) | GSA-Net · FBMS | JM: 76.1 -> 76.3 | Helped |
| Add KNN filtering to guided slots (ablation) | GSA-Net · DAVIS-16 | GM: 84.1 -> 86.1 | Helped |
| Add KNN filtering to guided slots (ablation) | GSA-Net · FBMS | JM: 76.3 -> 78.4 | Helped |
| Add FAT to guided slots, KNN filtering, and slot attention (ablation) | GSA-Net · DAVIS-16 | GM: 86.5 -> 87.7 | Helped |
| Add FAT to guided slots, KNN filtering, and slot attention (ablation) | GSA-Net · FBMS | JM: 78.9 -> 79.2 | Helped |
| Use GSA-Net* with MiT-b2 instead of HFAN* with MiT-b2 | unsupervised video object segmentation · DAVIS-16 val | GM: 88.7 -> 88.9 | Helped |
| Use GSA-Net* with MiT-b2 instead of HFAN* with MiT-b2 | unsupervised video object segmentation · DAVIS-16 val | JM: 88.0 -> 88.3 | Helped |
| Use GSA-Net* with MiT-b2 instead of HFAN* with MiT-b2 | unsupervised video object segmentation · DAVIS-16 val | FM: 89.3 -> 89.6 | Helped |
| Use GSA-Net* with MiT-b2 instead of OAST | unsupervised video object segmentation · FBMS | JM: 83.0 -> 83.1 | Helped |
| Increase slot-attention test iterations beyond 3 (ablation) | GSA-Net · DAVIS-16 | score | No clear effect |
Limitations
- Relies on an optical flow stream (RAFT) in addition to RGB, adding cost and dependence on flow quality.
- Evaluated only on DAVIS-16 and FBMS for unsupervised VOS.
Experimental setup
Datasets: DUTS DAVIS-16 YouTube-VOS FBMS
Benchmarks: DAVIS-16 FBMS
Metrics: J mean (region similarity) F mean (boundary accuracy) G (average of J and F) FPS
Baselines: MATNet RTNet FSNet TransportNet AMC-Net IMP HFAN PMN TMO OAST
Builds on: slot attention HFAN RAFT TMO
Closest papers by meaning
- SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers (CVPR 2024 · similarity 0.87)
- Dual Prototype Attention for Unsupervised Video Object Segmentation (CVPR 2024 · similarity 0.86)
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (CVPR 2024 · similarity 0.86)
- GLASS: Guided Latent Slot Diffusion for Object-Centric Learning (CVPR 2025 · similarity 0.86)
- Grounded Object-Centric Learning (ICLR 2024 · similarity 0.86)
- Learning Spatial-Semantic Features for Robust Video Object Segmentation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.