Research map / Vision-language and multimodal / Robot action and navigation
Learning Interactive Real-World Simulators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a universal action-conditioned video diffusion simulator (UniSim, 5.6B) from orchestrated diverse datasets, then uses it to train vision-language policies, RL low-level policies, and video captioners that transfer zero-shot or improve on real data.
Problem
Building a real-world interactive simulator that predicts visual outcomes of both high-level language actions and low-level controls is hindered because no single dataset covers objects, scenes, actions, motions and robot controls together.
Approach
Unify heterogeneous datasets (simulation, robot, human activity, panorama scans, internet text-image) into a common action space (T5 text embeddings plus discretized controls) and train a history-conditioned video diffusion model as an observation prediction model that can be rolled out autoregressively and used to train downstream agents.
Key results
- Simulator-Hindsight VLM achieves RDG 0.34 vs 0.07-0.11 for VLM-BC trained on short-horizon data on long-horizon Language Table tasks (3-4 times better).
- RL-finetuned VLA reaches 0.81 overall success vs 0.58 for BC and 0.71 vs 0.12 on pointing tasks in simulated rollouts.
- PaLI-X finetuned purely on UniSim-generated videos improves ActivityNet CIDEr from 15.2 to 46.23, reaching 84% of the performance of finetuning on real data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the VLM with simulated long-horizon hindsight data instead of original short-horizon data | PALM-E 12B VLM · Language Table simulated evaluation | RDG (moved): 0.11 $\pm$ 0.13 -> 0.34 ± 0.13 | Helped |
| Train the VLM with simulated long-horizon hindsight data instead of original short-horizon data | PALM-E 12B VLM · Language Table simulated evaluation | RDG (all): 0.07 $\pm$ 0.11 -> 0.34 ± 0.13 | Helped |
| Fine-tune the VLA policy with RL in UniSim | PaLI 3B VLA policy · Language Table simulated rollouts | Success rate (all): 0.58 -> 0.81 | Helped |
| Fine-tune the VLA policy with RL in UniSim | PaLI 3B VLA policy · Language Table simulated rollouts, pointing tasks | Success rate (pointing): 0.12 -> 0.71 | Helped |
| Condition on 4 recent frames instead of 1 frame (ablation) | UniSim · Ego4D validation | FVD: 315.69 -> 211.3 | Helped |
| Condition on 4 recent frames instead of 1 frame (ablation) | UniSim · Ego4D validation | FID: 59.47 -> 34.63 | Helped |
| Use 4 distant history frames instead of 4 recent frames (ablation) | UniSim · Ego4D validation | FVD: 211.3 -> 237 | Hurt |
| Increase conditioning history beyond 4 frames (ablation) | UniSim · Ego4D validation | performance | No clear effect |
| Remove internet data from training (ablation) | UniSim · Held-out test split | FVD: 211.30 -> 307.80 | Hurt |
| Remove internet data from training (ablation) | UniSim · Held-out test split | CLIP: 22.63 -> 21.99 | Hurt |
| Train on broad internet, human activity, and robot data rather than internet data only (ablation) | UniSim · Held-out test split | FVD: 219.62 -> 211.30 | Helped |
| Increase model size from 500M to 5.6B (ablation) | UniSim · Held-out test split | FVD: 277.85 -> 211.30 | Helped |
Surprising findings
- Adding a domain identifier to actions improves low-data domain quality in-distribution but hurts generalization to other domains.
- Model scaling plateaus: FVD gains diminish as model size grows, which the authors describe as slightly disappointing for scaling.
Limitations
- Hallucinates outcomes for unrealistic actions (e.g., wash hands on a tabletop robot) and cannot detect impossible actions.
- Limited memory from conditioning on few recent frames; objects can disappear when out of recent history.
- Limited out-of-domain generalization, mostly trained on 4 robot morphologies; visual-only simulation.
Experimental setup
Datasets: Habitat HM3D Language Table sim Language Table real Bridge Data RT-1 data Ego4D Something-Something V2 EPIC-KITCHENS Matterport Room-to-Room scans LAION-400M ALIGN ActivityNet Captions
Benchmarks: Ego4D validation (FVD, FID, IS, CLIP) Language Table simulated long-horizon evaluation (RDG) Language Table simulated RL rollouts (success rate) ActivityNet Captions (CIDEr) MSR-VTT VATEX SMIT
Metrics: FVD FID Inception score CLIP score reduction in distance to goal success rate CIDEr
Baselines: VLM-BC VLA-BC PaLI-X finetuned on true ActivityNet Captions Internet only (dataset ablation) Without internet (dataset ablation)
Builds on: diffusion models classifier-free guidance video U-Net PALM-E PaLI
Closest papers by meaning
- SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning (NeurIPS 2026 · similarity 0.89)
- Genie: Generative Interactive Environments (ICML 2024 · similarity 0.88)
- Exploration-Driven Generative Interactive Environments (CVPR 2025 · similarity 0.88)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.88)
- VirtualEnv: A Platform for Embodied AI Research (AAAI 2026 · similarity 0.88)
- Imagined Autocurricula (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.