Research map / Video / Video generation
RoboDreamer: Learning Compositional World Models for Robot Imagination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
RoboDreamer factorizes text-to-video robot world-model generation by parsing instructions into action and relation primitives, composing per-primitive diffusion scores, and adding goal image/sketch conditions for compositional generalization.
Problem
Text-to-video models used as robot world models fail to generalize to unseen combinations of language instructions and object relations, which are needed for planning in new environments.
Approach
Parse instructions into verb and prepositional phrase primitives and compose their diffusion score functions (product-of-experts style averaging of epsilon predictions), with a hybrid training objective over random subsets of components, extended to goal image and sketch conditions.
Key results
- On unseen RT-1 instructions, RoboDreamer reaches 81.3% human-rated task completion versus 46.9% for AVDC and 50.1% for HiP.
- With goal image conditioning, RoboDreamer (t+i) reaches 95.8 human score and 444.3 FVD versus AVDC's 46.9 and 517.1.
- On RLBench, RoboDreamer achieves 49.3% average success rate versus 44.2% for Hiveformer and 41.0% for UniPi.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text parsing to factorize task instructions (ablation) | RoboDreamer · RT-1 human evaluation, seen tasks | Human: 85.5 -> 90.1 | Helped |
| add text parsing to factorize task instructions (ablation) | RoboDreamer · RT-1 human evaluation, unseen tasks | Human: 68.8 -> 81.3 | Helped |
| add a goal sketch alongside language | RoboDreamer · RT-1 multimodal generation | Human: 81.3 -> 94.7 | Helped |
| add a goal sketch alongside language | RoboDreamer · RT-1 multimodal generation | FVD: 487.8 -> 454.7 | Helped |
| add a goal image alongside language | RoboDreamer · RT-1 multimodal generation | Human: 81.3 -> 95.8 | Helped |
| add a goal image alongside language | RoboDreamer · RT-1 multimodal generation | FVD: 487.8 -> 444.3 | Helped |
| use RoboDreamer instead of the strongest baseline | robot manipulation planning · RLBench average | Success rate: 44.2 -> 49.3 | Helped |
| use RoboDreamer instead of the strongest baseline | robot manipulation planning · RLBench, lift block | Success rate: 28.2 -> 22.2 | Hurt |
| use RoboDreamer instead of the strongest baseline | robot manipulation planning · RLBench, lamp on | Success rate: 53.2 -> 51.9 | Hurt |
Surprising findings
- Human-rated scores for the RoboDreamer variants are reported on a single small set of about 128 samples with at least three raters per sample, a limited evaluation scale.
Limitations
- Limited to single-camera views and cannot use multi-camera 3D information.
- Generalizes poorly to many real-world images outside the training distribution.
- Video generation is constrained in moving-camera settings.
Experimental setup
Datasets: RT-1 RLBench RT-X
Benchmarks: RT-1 held-out unseen instructions RLBench
Metrics: human task-completion score FVD IMO (GroundingDINO bounding box IoU) success rate
Baselines: AVDC HiP RoboDreamer w/o text parsing Image-BC Hiveformer UniPi
Builds on: AVDC Imagen UniPi HiP ControlNet
Closest papers by meaning
- Solving New Tasks by Adapting Internet Video Knowledge (ICLR 2025 · similarity 0.89)
- Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination (ICLR 2025 · similarity 0.89)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.89)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.89)
- RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.89)
- Video Language Planning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.