Research map / Architectures and efficiency / RNNs and normalization
Parallelized Spatiotemporal Slot Binding for Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSB, a temporally parallelizable slot-binding encoder using causal cross-attention and time/object-axis self-attention, evaluated as an encoder in 2D video and 3D multi-camera object-centric autoencoders.
Problem
Object-centric slot encoders for video are RNN-based, causing training instability on long sequences, linear-in-length training time, and poor capacity.
Approach
Replace recurrence with a fixed stack of PSB blocks that refine initial slots for all time-steps in parallel, using causal cross-attention to input features, per-slot time-axis self-attention, and per-time-step slot self-attention.
Key results
- On MOVi-A/B with spatial broadcast decoder, PSB improves video FG-ARI and PSNR over SAVi and remains stable at 12-frame training episodes where SAVi diverges.
- PSB gives about 1.6x faster training step than SAVi on long episodes.
- With the mixture decoder on 2D videos, reported improvements of 14.7-26.8% FG-ARI and 2.9-7.6% PSNR over the recurrent baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace recurrent slot binding with PSB using an alpha-mixture decoder | SAVi encoder · MOVi-A and MOVi-B | FG-ARI: 14.7-26.8% in FG-ARI | Helped |
| Replace recurrent slot binding with PSB using an alpha-mixture decoder | SAVi encoder · MOVi-A and MOVi-B | PSNR: 2.9-7.6% in reconstruction PSNR | Helped |
| Replace recurrent slot binding with PSB in dynamic 3D scenes | SAVi encoder with NeRF decoder · Dynamic 3D CLEVR-Simple and Dynamic 3D CLEVR-Natural-Ego | slot linear-probing performance: 7.3-121% in slot linear-probing performance | Helped |
| Replace recurrent slot binding with PSB in dynamic 3D scenes | SAVi encoder with NeRF decoder · Dynamic 3D CLEVR-Simple and Dynamic 3D CLEVR-Natural-Ego | PSNR: 4-8% in PSNR | Helped |
| Use PSB instead of the recurrent encoder with an autoregressive transformer decoder | STEVE · MOVi-C, MOVi-D, and MOVi-E | FG-ARI | Mixed |
| Increase training episode length from 6 to 12 | SAVi and PSB · MOVi-A and MOVi-B | training stability | Mixed |
| Initialize slots by randomly sampling them from a learned Gaussian instead of using learned parameters (ablation) | PSB · 2D and 3D experiments | overall performance | Hurt |
| Use joint self-attention over all NT slots instead of decoupled time-axis and object-axis self-attention (ablation) | PSB · 2D and 3D experiments | overall performance | No clear effect |
| Remove inverted attention and renormalization from bottom-up attention (ablation) | PSB · Video object decomposition | FG-ARI | Hurt |
Surprising findings
- On visually complex MOVi-C/D/E with an autoregressive transformer decoder, segmentation is slightly worse than STEVE though the difference is not substantial.
Limitations
- Attention-based design has quadratic memory complexity in sequence length.
- Evaluated only on synthetic datasets; not applied to longer episodes or real scenes.
Experimental setup
Datasets: MOVi-A MOVi-B MOVi-C MOVi-D MOVi-E Dynamic 3D CLEVR-Simple Dynamic 3D CLEVR-Natural-Ego
Benchmarks: MOVi-A MOVi-B MOVi-C MOVi-D MOVi-E Dynamic 3D CLEVR-Simple Dynamic 3D CLEVR-Natural-Ego
Metrics: video FG-ARI reconstruction PSNR slot linear probing R2/accuracy novel view synthesis PSNR training step latency
Baselines: SAVi STEVE uORF OSRT Slot Attention per time-step
Builds on: Slot Attention SAVi STEVE OSRT uORF
Closest papers by meaning
- Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning (CVPR 2026 · similarity 0.88)
- Slot State Space Models (NeurIPS 2024 · similarity 0.87)
- GLASS: Guided Latent Slot Diffusion for Object-Centric Learning (CVPR 2025 · similarity 0.86)
- Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation (ICLR 2025 · similarity 0.86)
- Slot-VLM: Object-Event Slots for Video-Language Modeling (NeurIPS 2024 · similarity 0.86)
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.