Research map / Video / Video segmentation and tracking
Dual Prototype Attention for Unsupervised Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes prototype-based inter-modality attention (IMA) and inter-frame attention (IFA) for fusing RGB and optical-flow cues and aggregating global video context in unsupervised video object segmentation, reaching state-of-the-art on DAVIS 2016, FBMS and YouTube-Objects.
Problem
Unsupervised video object segmentation needs effective fusion of appearance and motion cues and temporal aggregation across frames, without the high cost of iterative temporal refinement or noisy naive fusion.
Approach
Prototype-based cross-attention that densely exchanges information between RGB and flow branches (IMA) and propagates global context from uniformly sampled reference frames stored in a memory to each query frame (IFA), with prototypes built via channel-wise softmax soft-region pooling inspired by OCR.
Key results
- DPA achieves 86.9 GM on DAVIS 2016 at 352x352 and 87.6 GM at 512x512, outperforming all compared methods at the same resolution.
- On FBMS test set DPA reaches 83.4 JM at 512x512 (81.2 at 352x352), about 1.5 points above the next best at the same resolution.
- On YouTube-Objects DPA reports a mean J of 73.7, above TMO's 71.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IMA with prototype embedding to the baseline (ablation) | DPA · DAVIS 2016 validation set | GM: 83.4 -> 85.9 | Helped |
| Add IFA with prototype embedding to the baseline (ablation) | DPA · DAVIS 2016 validation set | GM: 83.4 -> 85.4 | Helped |
| Add IMA and IFA with prototype embedding (ablation) | DPA · DAVIS 2016 validation set | GM: 83.4 -> 86.9 | Helped |
| Use more IFA reference frames, from 1 to 5 (ablation) | DPA · DAVIS 2016 validation set | GM: 86.3 -> 86.9 | Mixed |
| Add prototype embedding to IMA (ablation) | DPA · DAVIS 2016 validation set | GM: 86.1 -> 86.9 | Helped |
| Add prototype embedding to IFA (ablation) | DPA · DAVIS 2016 validation set | GM: 86.3 -> 86.9 | Helped |
| Add prototype embedding to both IMA and IFA (ablation) | DPA · DAVIS 2016 validation set | GM: 85.3 -> 86.9 | Helped |
| Use DPA instead of OAST | DPA · FBMS test set | JM: 81.9 -> 81.2 | Hurt |
| Use DPA instead of HFAN | DPA · DAVIS 2016 validation set | GM: 87.0 -> 86.9 | No clear effect |
| Use DPA instead of TMO | DPA · YouTube-Objects | J mean: 71.5 -> 73.7 | Helped |
Limitations
- Inference time increases with IMA and IFA (0.0164 s to 0.0414 s per frame), and gains from more reference frames saturate beyond three.
- Relies on pre-computed optical flow from RAFT, adding a separate preprocessing model.
Experimental setup
Datasets: DUTS DAVIS 2016 YouTube-VOS 2018
Benchmarks: DAVIS 2016 FBMS YouTube-Objects
Metrics: region similarity J boundary accuracy F G (average of J and F)
Baselines: TMO PMN HFAN MATNet RTNet FSNet AMC-Net OAST IMP COSNet
Builds on: OCR TMO PMN F2Net RAFT
Closest papers by meaning
- Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation (AAAI 2024 · similarity 0.88)
- VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE (ICLR 2024 · similarity 0.86)
- Guided Slot Attention for Unsupervised Video Object Segmentation (CVPR 2024 · similarity 0.86)
- M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (CVPR 2025 · similarity 0.86)
- Audio-Visual Segmentation via Unlabeled Frame Exploitation (CVPR 2024 · similarity 0.86)
- Event-assisted Low-Light Video Object Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.