Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreMa, a compositional manipulation world model built from object-centric Gaussian Splatting and the PyBullet physics engine, which generates equivariantly transformed demonstrations to improve one-shot imitation learning for PerAct.
Problem
Imitation learning policies need many demonstrations and generalize poorly to new object configurations; existing world models are statistical and hallucinate, so they are unsuitable for grounded robot imagination.
Approach
Treat the world model as a learnable digital twin: reconstruct object-centric assets with Gaussian Splatting, convert them to meshes for PyBullet dynamics, and apply rigid roto-translations to object poses and end-effector actions to imagine new valid demonstrations, verified by goal-position thresholds.
Key results
- In single-task RLBench, DreMa + Original averages 25.1% vs 16.0% for PerAct with original data (about +9.1 points).
- In multi-task RLBench on close jar, slide block, and sort shape, DreMa with all data reaches 35.5% average vs 22.4% for PerAct.
- On a real Franka Panda across five tasks, DreMa (All) averages 62.9% in-distribution-and-OOD-combined vs 31.7% for PerAct, with OOD average 58.3% vs 25.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train PerAct with imagination demonstrations instead of original demonstrations | PerAct · RLBench single-task, average | success rate: 16.0 -> 19.1; +9.1% | Helped |
| Combine imagination and original demonstrations | PerAct · RLBench single-task, average | success rate: 16.0 -> 25.1 | Helped |
| Train PerAct with imagination demonstrations instead of original demonstrations | PerAct · RLBench multi-task, average | success rate: 22.4 -> 31.3; +9.1 | Helped |
| Combine imagination and original demonstrations | PerAct · RLBench multi-task, average | success rate: 22.4 -> 35.5; +13.1% | Helped |
| Combine imagination and original demonstrations | PerAct · Real robot, average in-distribution | success rate: 31.7 -> 62.9; 62.9% vs 31.7% | Helped |
| Combine imagination and original demonstrations | PerAct · Real robot, average OOD | success rate: 25.0 -> 58.3 | Helped |
| Use imagination demonstrations only instead of original demonstrations | PerAct · Real robot pick block, 4 examples | success rate: 55 -> 50 | Hurt |
| Use imagination demonstrations only instead of original demonstrations | PerAct · Real robot pick block, 8 examples | success rate: 55 -> 80; +25% | Helped |
| Add object rotation augmentation (ablation) | DreMa · RLBench close jar | success rate: 10.0 ± 3.10 -> 25.2 ± 0.98 | Helped |
| Add roto-translation augmentation (ablation) | DreMa · RLBench close jar | success rate: 10.0 ± 3.10 -> 41.2 ± 2.40 | Helped |
| Combine object rotation and roto-translation with original demonstrations (ablation) | DreMa · RLBench close jar | success rate: 41.2 ± 2.40 -> 51.2 ± 1.60 | Helped |
| Generate imagination demonstrations from 20 original examples rather than train on the original examples alone | DreMa · RLBench sort shape and slide block | success rate: 6.4%; 10.4% | Helped |
Surprising findings
- Imagined-only data trained PerAct outperformed PerAct on original data in the simulated tasks (+3.1 single-task, +9.1 multi-task), indicating imagination alone is useful.
- On the real pick block task with only four original examples, imagination-only training performed worse than the original-data policy (50 vs 55).
Limitations
- Requires full observability of the environment and handles only simple, rigid objects (no articulation or deformation).
- Open-loop construction without feedback; errors in segmentation and mesh prediction propagate to dynamics and collisions.
- Gains are modest on some tasks and the real-world evaluation uses few trials (20 configurations per task).
Experimental setup
Datasets: RLBench simulated demonstrations real Franka Panda demonstrations YCB object scans
Benchmarks: RLBench (close jar, insert peg, lift, pick cup, sort shape, place wine, put in cupboard, slide block, stack blocks) real Franka Panda in- and out-of-distribution tasks
Metrics: success rate (mean ± std and max) localization error (m)
Baselines: PerAct Random patches Random table color Distractors MimicGen
Builds on: 3D Gaussian Splatting 2D Gaussian Splatting PerAct DEVA PyBullet
Closest papers by meaning
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.89)
- DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025 · similarity 0.88)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.88)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.88)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.87)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.