Research map / Video / Video-language understanding
Slot State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SlotSSMs, a state space model whose state is a set of independently transitioned slots with sparse cross-slot interaction via self-attention, applied to object-centric learning, long-context video reasoning, and 3D visual reasoning.
Problem
Standard SSMs keep a monolithic state vector, which entangles the dynamics of independent entities in modular sequences such as multi-object videos; RNN-based slot models lack parallel training and struggle with long-range dependencies.
Approach
Maintain the SSM state as K slots, each transitioned by block-diagonal (Mamba-style) SSM parameters conditioned only on that slot, with inter-slot interaction introduced by a self-attention slot mixer and slot encoders that extract slots from unstructured inputs; OC-SlotSSM adds inverted attention for object-centric decomposition.
Key results
- On CATER snitch localization with pre-training, OC-SlotSSM reaches 69.27% top-1 vs 49.21% for SlotTransformer and 41.15% for Single State SSM.
- On Blinking Color Balls, SlotSSM maintains consistent performance from sequence length 80 to 2560 while baselines degrade or fail to train at long lengths.
- OC-SlotSSM outperforms SAVi on unsupervised segmentation and attribute prediction on MOVi-A and MOVi-B and avoids the scene-collapse instability seen in SAVi.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SlotSSM instead of a monolithic Single State SSM | CATER Snitch Localization · CATER Snitch Localization (No Pre-train) | Top-1 Acc (%): 10.27 -> 25.64 | Helped |
| Use SlotSSM instead of SlotTransformer | CATER Snitch Localization · CATER Snitch Localization (No Pre-train) | Top-1 Acc (%): 41.09 -> 25.64 | Hurt |
| Use SlotSSM instead of SlotTransformer | CATER Snitch Localization · CATER Snitch Localization (Pre-train) | Top-1 Acc (%): 49.21 -> 54.73 | Helped |
| Add pre-training | SlotSSM on CATER Snitch Localization · CATER Snitch Localization | Top-1 Acc (%): 25.64 -> 54.73 | Helped |
| Use OC-SlotSSM instead of SlotSSM | CATER Snitch Localization · CATER Snitch Localization (No Pre-train) | Top-1 Acc (%): 25.64 -> 61.58 | Helped |
| Use OC-SlotSSM instead of SlotSSM | CATER Snitch Localization · CATER Snitch Localization (Pre-train) | Top-1 Acc (%): 54.73 -> 69.27 | Helped |
| Use OC-SlotSSM instead of SAVi++ | Depth estimation · UT Egocentric | Depth Estimation MSE: 0.589 -> 0.464 | Helped |
| Use OC-SlotSSM instead of SAVi++ | Depth estimation · Waymo | Depth Estimation MSE: 0.804 -> 0.653 | Helped |
| Use OC-SlotSSM instead of SAVi++ | Depth estimation · TikTok | Depth Estimation MSE: 1.412 -> 1.180 | Helped |
Surprising findings
- RIM fails to generalize on the Blinking Color Balls benchmark at any sequence length, attributed to optimization issues from separate per-slot weights.
- Without pre-training, SlotSSM underperforms SlotTransformer on CATER (25.64% vs 41.09% top-1), while pre-training reverses this ordering.
Limitations
- SlotSSMs benefit more from pre-training than Transformers and underperform them without it on sparse-signal visual reasoning
- Not scaled to industry-size models or data due to compute constraints
- Only evaluated on video and visual tasks; text and audio left unexplored
Experimental setup
Datasets: bouncing balls MOVi-A MOVi-B CATER Blinking Color Balls (constructed) TikTok Dancing UT Egocentric Waymo Open Dataset
Benchmarks: bouncing balls video prediction Blinking Color Balls Benchmark MOVi-A MOVi-B CATER snitch localization UT Egocentric depth Waymo depth TikTok depth
Metrics: MSE FG-ARI mIoU top-1 accuracy top-5 accuracy R^2 inference latency
Baselines: Single State SSM SlotTransformer RIM Transformer SAVi SAVi++ SlotRNN
Builds on: Mamba Slot Attention SAVi RIM S4
Closest papers by meaning
- Slot-VLM: Object-Event Slots for Video-Language Modeling (NeurIPS 2024 · similarity 0.88)
- Parallelized Spatiotemporal Slot Binding for Videos (ICML 2024 · similarity 0.87)
- Cycle Consistency Driven Object Discovery (ICLR 2024 · similarity 0.87)
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (CVPR 2024 · similarity 0.86)
- StateSpaceDiffuser: Bringing Long Context to Diffusion World Models (NeurIPS 2025 · similarity 0.85)
- MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.