Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Learning 3D Persistent Embodied World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a persistent embodied world model by conditioning a CogVideoX-based RGB-D video diffusion model on a volumetric DINO-feature 3D memory map, enabling consistent long-horizon simulation, planning, and policy learning.
Problem
Video-based embodied world models are myopic: they lack memory of unobserved or previously generated scene content, so they generate contradictions and cannot support consistent long-horizon planning in partially observed 3D environments.
Approach
Fill a volumetric 3D grid with DINO-v2 features unprojected from generated RGB-D frames and inject it into the video diffusion model via cross-attention memory blocks, with Plücker camera embeddings concatenated to the latents for pixel-wise action control; memory blocks are trained in a second stage with the backbone frozen.
Key results
- Ours reaches FVD 91.9 and PSNR 22.46 versus NWM FVD 194.0 and PSNR 17.48 on video generation.
- Ranking NoMaD trajectories, Ours gets ATE 4.47 and RPE 3.28 versus NWM 4.54 and 3.51 at the larger sample size.
- Ours with MPC reaches SIM 87.5 after about 720 iterations, described as competitive with NoMaD-based ranking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3D memory to the video model instead of using NWM without memory | Persistent Embodied World Model · Video generation | FVD: 194.040\systd{4.313} -> 91.885\systd{2.033} | Helped |
| Use persistent 3D memory instead of Image Memory | Persistent Embodied World Model · Video generation | PSNR: 18.977\systd{0.248} -> \textbf{22.458\systd{ 0.052}} | Helped |
| Add depth information to the RGB video model (ablation) | Persistent Embodied World Model · Video generation | visual quality and consistency | Helped |
| Use a 3D feature map instead of a 2D feature map (ablation) | Persistent Embodied World Model · Video generation | visual quality and consistency | Helped |
| Add 3D memory to NWM for trajectory ranking | Persistent Embodied World Model · Trajectory ranking | SIM: 59.7\systd{3.5} -> 70.8\systd{1.7} | Helped |
Limitations
- Requires training data with depth, limiting use on most real-world datasets.
- The 3D memory does not model dynamic changes in the environment over time.
- Evaluated mainly in Habitat simulation, whose visual quality is limited.
Experimental setup
Datasets: HM3D Habitat simulation trajectories
Benchmarks: Habitat HM3D held-out scenes
Metrics: PSNR SSIM LPIPS FVD DreamSim SRC ATE RPE SIM
Baselines: NWM Image Memory (3D-Mem) NoMaD Ours w/o depth Ours w/ 2D-Map
Builds on: CogVideoX Navigation World Models DINOv2 Plücker embedding NoMaD
Closest papers by meaning
- Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.90)
- DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning (NeurIPS 2026 · similarity 0.89)
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.89)
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.89)
- WorldMem: Long-term Consistent World Simulation with Memory (NeurIPS 2025 · similarity 0.89)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.