Research map / Video / Video segmentation and tracking
VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VONet, an unsupervised video object learning model that pairs a parallel U-Net attention network (all slot masks generated at once) with an object-wise sequential VAE prior, reaching state-of-the-art FG-ARI and mIoU on five MOVI datasets.
Problem
Unsupervised video object learning needs fine-grained yet temporally consistent slot masks, and MONet-style recurrent attention is too slow for many slots and was never extended to video.
Approach
Replace MONet's sequential scope-based attention with a parallel U-Net that generates all K masks at once, with a mask transformer for inter-slot communication, and carry slot context vectors across frames; add a sequential VAE whose learned prior forecasts per-slot latents from all slots' past states, with a KL term encouraging temporal predictability.
Key results
- VONet reaches FG-ARI 91.0 on MOVI-A and 60.6 on MOVI-B, beating the best baseline (SCALOR 68.1 and 45.3) on all five MOVI datasets.
- On mIoU, VONet achieves 60.5, 59.7, 42.8, 37.7, 36.4 across MOVI-A to E, exceeding STEVE, the strongest mIoU baseline on MOVI-C to E (30.5, 23.8, 26.2).
- Parallel attention keeps near-constant inference time as slots grow, whereas MONet's time scales linearly with slot count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-A | FG-ARI: 68.1{\scriptstyle \pm 1.4} -> 91.0{\scriptstyle \pm 1.5} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-A | mIoU: 59.8{\scriptstyle \pm 0.9} -> 60.5{\scriptstyle \pm 2.0} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-B | FG-ARI: 45.3{\scriptstyle \pm 0.6} -> 60.6{\scriptstyle \pm 0.8} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-B | mIoU: 46.6{\scriptstyle \pm 0.1} -> 59.7{\scriptstyle \pm 2.9} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-C | FG-ARI: 38.1{\scriptstyle \pm 0.3} -> 45.3{\scriptstyle \pm 0.4} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-C | mIoU: 30.5{\scriptstyle \pm 0.2} -> 42.8{\scriptstyle \pm 0.5} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-D | FG-ARI: 42.9{\scriptstyle \pm 2.8} -> 50.7{\scriptstyle \pm 1.1} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-D | mIoU: 23.8{\scriptstyle \pm 3.7} -> 37.7{\scriptstyle \pm 0.3} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-E | FG-ARI: 49.7{\scriptstyle \pm 1.0} -> 56.3{\scriptstyle \pm 0.5} | Helped |
| Use VONet instead of the strongest listed baseline | Unsupervised video object learning · MOVI-E | mIoU: 26.2{\scriptstyle \pm 1.3} -> 36.4{\scriptstyle \pm 0.7} | Helped |
| Add U-Net mask refinement (ablation) | VONet · MOVI-C, MOVI-D, MOVI-E | segmentation performance | Helped |
| Add replay-buffer training from past slot states (ablation) | VONet · MOVI-A, MOVI-B | long-term object tracking | Helped |
Surprising findings
- STEVE performs well on the complex MOVI-C/D/E datasets but has unexpectedly low FG-ARI on the simpler MOVI-A/B because of over-segmentation and identity swaps with redundant slots.
- Removing the KL term entirely causes unstable training and eventual crashes, while without KL balancing VONet fails to learn meaningful masks on MOVI-A.
Limitations
- Over-segmentation when the scene has fewer objects than pre-allocated slots, with background fragmentation.
- Incomplete object understanding (multi-texture objects split or merged) due to the lack of objectness priors.
- Short-term GRU memory may expire under long occlusion, and the KLD term can stay low even when a slot loses temporal consistency.
Experimental setup
Datasets: MOVI-A MOVI-B MOVI-C MOVI-D MOVI-E
Benchmarks: MOVI-A MOVI-B MOVI-C MOVI-D MOVI-E
Metrics: FG-ARI mIoU
Baselines: SAVI SIMONe SCALOR ViMON STEVE MONet
Builds on: MONet STEVE SAVI ViMON SIMONe
Closest papers by meaning
- Unsupervised Learning From Video With Deep Neural Embeddings (CVPR 2020 · similarity 0.87)
- Learning Unsupervised Video Object Segmentation Through Visual Attention (CVPR 2019 · similarity 0.87)
- Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos (NeurIPS 2022 · similarity 0.87)
- Dense Unsupervised Learning for Video Segmentation (NeurIPS 2021 · similarity 0.87)
- Dual Prototype Attention for Unsupervised Video Object Segmentation (CVPR 2024 · similarity 0.86)
- Putting the Object Back into Video Object Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.