Research map / Video / Video segmentation and tracking
Dreamweaver: Learning Compositional World Models from Pixels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dreamweaver, a recurrent block-slot architecture trained with a multi-future-frame predictive objective to learn static and dynamic compositional concepts from raw videos and enable compositional video imagination without language.
Problem
Unsupervised learning of compositional world models from raw video: discovering reusable object and attribute concepts (both static and dynamic) without text, masks, or boxes, and recombining them to imagine novel futures.
Approach
A Recurrent Block-Slot Unit (RBSU) keeps slots subdivided into blocks with per-block GRU/MLP and prototype-attention attractor dynamics, trained end-to-end with a predictive (future-frame) cross-entropy loss over dVAE tokens rather than reconstruction, so dynamic factors such as direction and speed emerge as blocks.
Key results
- Dreamweaver achieves the highest DCI disentanglement, completeness and informativeness across all datasets versus RSSM, STEVE and SysBinder.
- On dynamic-factor informativeness (I-D), Dreamweaver scores more than twice the other baselines.
- On Dancing-CLEVR, all baselines fail at frame offset zero in the scene prediction probe while Dreamweaver remains accurate; Dreamweaver also generalizes to OOD Dancing-Sprites combinations where STEVE and RSSM degrade.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Dreamweaver's predictive objective instead of reconstruction training (ablation) | Dreamweaver · Dancing-CLEVR | Informativeness-Dynamic (I-D) | Helped |
| Add concept memory (ablation) | Dreamweaver · Dancing-CLEVR | Disentanglement (D) | Helped |
| Use concept memory after every slot iteration rather than only on the last slot iteration (ablation) | Dreamweaver · Dancing-CLEVR | Informativeness-Dynamic (I-D) | Mixed |
| Increase the number of blocks until it is large enough for the dataset (ablation) | Dreamweaver · CLEVR-Hard | Overall performance | Helped |
| Increase concept memory size (ablation) | Dreamweaver · CLEVR-Hard | Disentanglement | Mixed |
| Increase context length (ablation) | Dreamweaver · Dancing-CLEVR and Dancing-Sprites | Disentanglement and informativeness | Mixed |
| Use context length below two frames (ablation) | Dreamweaver · Dancing-Sprites | Overall performance | Hurt |
| Increase context length beyond four frames (ablation) | Dreamweaver · Dancing-Sprites | Overall performance | Hurt |
| Increase prediction length beyond two frames (ablation) | Dreamweaver · Dancing-CLEVR and Dancing-Sprites | Overall performance | Hurt |
| Use Dreamweaver's predictive objective rather than reconstruction training | Dreamweaver · Dancing-CLEVR downstream prediction | Prediction accuracy | Helped |
| Use block-slot representations rather than monolithic or object-slot representations | Dreamweaver · DCI across five video datasets | Informativeness-Dynamic (I-D) | Helped |
| Use block-slot representations for downstream OOD generalization | Dreamweaver and SysBinder · Dancing-Sprites-OOD | Prediction accuracy | Helped |
Surprising findings
- Baseline models such as SysBinder and STEVE fail even at frame offset zero on Dancing-CLEVR, showing reconstruction-trained block-slot models do not capture dynamic factors.
Limitations
- Not yet applicable to highly complex scene images, a general limitation of object-centric learning.
- Model is deterministic; extending to a probabilistic model is left as future work.
- Predicting more than 2 frames degrades performance, attributed to limited decoder capacity.
Experimental setup
Datasets: Moving-Sprites Dancing-Sprites Moving-CLEVR Dancing-CLEVR Moving-CLEVRTex Moving-Sprites-OOD
Benchmarks: DCI framework (Disentanglement, Completeness, Informativeness, Informativeness-Dynamic) Compositional scene prediction and reasoning probe on Dancing-Sprites and Dancing-CLEVR Compositional imagination MSE/LPIPS/PSNR on Moving-Sprites and Dancing-CLEVR
Metrics: DCI disentanglement DCI completeness DCI informativeness informativeness-dynamic probe accuracy MSE LPIPS PSNR
Baselines: RSSM STEVE SysBinder
Builds on: Slot Attention SysBinder DreamerV1 DCI framework dVAE
Closest papers by meaning
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.88)
- WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.87)
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.86)
- Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.86)
- Neurosymbolic Grounding for Compositional World Models (ICLR 2024 · similarity 0.86)
- Zero-shot World Models via Search in Memory (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.