Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Navigation World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a 1B-parameter Conditional Diffusion Transformer (CDiT) world model on egocentric robot and human videos with navigation actions, and uses it for MPC-style planning and for ranking NoMaD policy trajectories.
Problem
Visual navigation policies are fixed after training and cannot easily incorporate new constraints or allocate more compute at test time; the paper asks whether a video world model can simulate navigation trajectories for planning and ranking.
Approach
A Conditional Diffusion Transformer whose cross-attention-only conditioning on past frames gives complexity linear in context length, trained across robot and human egocentric videos with action and time-shift conditioning, then used to score imagined trajectories by perceptual similarity to a goal.
Key results
- NWM planning achieves ATE 1.13 and RPE 0.35 on RECON (2-second trajectories), outperforming NoMaD (ATE 1.93, RPE 0.52) and GNM (ATE 1.87, RPE 0.73).
- CDiT matches or beats DiT with up to 4x fewer FLOPs; CDiT-L is about 4x faster than DiT-XL while performing better.
- Adding unlabeled Ego4D video improves 4-second prediction on the unseen Go Stanford environment in LPIPS (0.658 to 0.652 for LPIPS, lower is better) and DreamSim (0.478 to 0.464 for DreamSim, lower is better) on the unseen environment; on RECON, the reported metrics show a drop, so the gain is limit
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Plan trajectories from scratch with NWM | NWM · RECON | ATE: 1.87 ± 0.00 -> 1.13 ± 0.02 | Helped |
| Plan trajectories from scratch with NWM | NWM · RECON | RPE: 0.73 ± 0.00 -> 0.35 ± 0.01 | Helped |
| Rank NoMaD trajectories using NWM | NoMaD · RECON | ATE: 1.93 ± 0.04 -> 1.78 ± 0.03 | Helped |
| Rank NoMaD trajectories using NWM | NoMaD · RECON | RPE: 0.52 ± 0.00 -> 0.48 ± 0.01 | Helped |
| Use 4 goals per sample instead of 1 (ablation) | NWM · RECON, 4 seconds ahead | LPIPS: 0.312 ± 0.001 -> 0.296 ±0.002 | Helped |
| Use 4 context frames instead of 1 (ablation) | NWM · RECON, 4 seconds ahead | LPIPS: 0.304 ± 0.001 -> 0.296 ±0.002 | Helped |
| Add action conditioning to time conditioning (ablation) | NWM · RECON, 4 seconds ahead | LPIPS: 0.760 ± 0.001 -> 0.295 ±0.002 | Helped |
| Add time conditioning to action conditioning (ablation) | NWM · RECON, 4 seconds ahead | LPIPS: 0.318 ± 0.002 -> 0.295 ±0.002 | Helped |
| Train with unlabeled Ego4D videos | NWM · Go Stanford, 4 seconds ahead | LPIPS: 0.658 ± 0.002 -> 0.652 ± 0.003 | Helped |
| Train with unlabeled Ego4D videos | NWM · RECON, 4 seconds ahead | LPIPS: 0.295 ±0.002 -> 0.368 ± 0.003 | Hurt |
| Train with unlabeled Ego4D videos | NWM · Go Stanford, 4 seconds ahead | PSNR: 11.031 ± 0.036 -> 11.083 ± 0.064 | Helped |
| Train with unlabeled Ego4D videos | NWM · RECON, 4 seconds ahead | PSNR: 15.343 ±0.060 -> 14.072 ± 0.075 | Hurt |
Surprising findings
- Training with Ego4D unlabeled video (time-shift action only) improves prediction on unseen Go Stanford, while the paper notes the model hallucinates paths in unknown environments.
Limitations
- Mode collapse and context loss when applied to out-of-distribution environments.
- Struggles to simulate temporal dynamics such as pedestrian motion.
- Limited to 3 DoF navigation actions.
Experimental setup
Datasets: SCAND TartanDrive RECON HuRoN Ego4D GO Stanford
Benchmarks: RECON GO Stanford
Metrics: LPIPS DreamSim PSNR FID FVD ATE RPE
Baselines: DIAMOND GNM NoMaD
Builds on: DiT DIAMOND NoMaD Stable Diffusion VAE Cross-Entropy Method
Closest papers by meaning
- EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.88)
- DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning (NeurIPS 2026 · similarity 0.88)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.88)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.88)
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction (NeurIPS 2025 · similarity 0.88)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.