Research map / Detection and segmentation / Open-set and few-shot detection
Dyn-O: Building Structured World Models with Object-Centric Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Dyn-O builds an object-centric world model on Cosmos-tokenizer slot features (with SAM2 masks used only during training via a dropout schedule), using an SSM for dynamics and splitting slots into static and dynamic components, evaluated on seven Procgen games.
Problem
Object-centric world models have only been validated in simple synthetic environments; it is unclear whether they can be learned from pixel trajectories in visually complex settings with diverse textures and cluttered scenes.
Approach
Learn slot representations on top of a frozen pretrained Cosmos tokenizer with SAM2 masks as attention guidance that are annealed away during training, then model slot dynamics with a shared SSM (Mamba) and adversarially disentangle each slot into static and dynamic features.
Key results
- Averaged across environments, Dyn-O achieves LPIPS 0.33 and FVD 361.3 versus DreamerV3's 0.42 and 692.5 in 20-step rollouts on unseen levels.
- Slot-object binding accuracy averages 0.51 for Dyn-O versus 0.33 for SOLV and 0.74 for the mask-always Oracle.
- Linear probing on coinrun shows static features at 34.9% and 37.7% for x/y position, near random-feature levels (29.3%, 28.3%), while dynamic features reach 78.1% and 75.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace monolithic DreamerV3 representation with object-centric slots in Dyn-O | Dyn-O vs DreamerV3 · Procgen 7 environments, 20-step rollout | FVD: 692.5 -> 361.3 | Helped |
| use object-centric representations instead of 14x14 patch-level features (ablation) | Dyn-O · Procgen 7 environments, 20-step rollout | FVD: 538.4 -> 361.3 | Helped |
| use object-centric representations instead of 14x14 patch-level features (ablation) | Dyn-O · Procgen 7 environments, 20-step rollout | LPIPS: 0.41 -> 0.33 | Helped |
| use object-centric representations instead of 14x14 patch-level features (ablation) | Dyn-O · Procgen 7 environments, 20-step rollout | SSIM: 0.53 -> 0.62 | Helped |
| use object-centric representations instead of 14x14 patch-level features (ablation) | Dyn-O · Procgen 7 environments, 20-step rollout | PSNR: 16.10 -> 16.34 | Helped |
| guide slot extraction with segmentation masks during training, then phase out mask use | Dyn-O representation encoder · Procgen average across four reported environments | FR-ARI: 0.33 -> 0.51 | Helped |
| keep segmentation masks active during both training and inference rather than use the mask-dropout schedule (ablation) | Dyn-O representation encoder · Procgen average across four reported environments | FR-ARI: 0.51 -> 0.74 | Helped |
| use object-centric slots instead of DreamerV3's monolithic representation | Dyn-O vs DreamerV3 · caveflyer, coinrun, ninja | LPIPS: caveflyer: 0.53 -> 0.61; coinrun: 0.34 -> 0.28; ninja: 0.48 -> 0.45 | Mixed |
| use object-centric slots instead of DreamerV3's monolithic representation | Dyn-O vs DreamerV3 · Procgen environments | PSNR: bigfish: 19.34 -> 20.28; caveflyer: 11.99 -> 10.82; coinrun: 14.22 -> 13.56; dodgeball: 16.10 -> 20.88; ninja: 12.58 -> | Mixed |
Surprising findings
- Dyn-O's PSNR is not uniformly better than DreamerV3 across environments; for example, DreamerV3 has higher PSNR on caveflyer, coinrun and ninja.
- Mask-free Dyn-O has lower slot-object binding accuracy than the always-on-mask Oracle, with the gap largest in dodgeball (0.48 vs 0.79).
Limitations
- Decoding is via a pretrained visual decoder rather than a more sophisticated generative decoder, limiting visual quality of rollouts.
- Object-slot binding remains imperfect without masks (for example, 0.27 on coinrun and 0.48 on dodgeball), and the slot count K is a fixed hyperparameter.
Experimental setup
Datasets: Procgen offline transitions (1M per environment)
Benchmarks: Procgen
Metrics: LPIPS FVD SSIM PSNR FG-ARI slot-object binding accuracy (FR-ARI) linear probing accuracy
Baselines: DreamerV3 Dyn-O w/o OC SOLV Oracle (always-on SAM2 masks)
Builds on: SOLV SlotSSM Slot Attention DreamerV3 Cosmos tokenizer
Closest papers by meaning
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (ICML 2025 · similarity 0.87)
- Zero-shot World Models via Search in Memory (NeurIPS 2025 · similarity 0.87)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.87)
- DyMoDreamer: World Modeling with Dynamic Modulation (NeurIPS 2025 · similarity 0.86)
- Learning Visual Feature-Based World Models via Residual Latent Action (NeurIPS 2026 · similarity 0.86)
- stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.