Research map / Multi-agent systems and games / Human-AI and goal-conditioned
RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
RoboGen uses LLMs (GPT-4), VLMs (Gemini-Pro) and generative models to propose tasks, build simulation scenes, generate training supervision and learn skills via RL, motion planning, or trajectory optimization in a fully generative propose-generate-learn loop.
Problem
Building simulated robot learning environments and training supervision (tasks, scenes, rewards) requires heavy human effort, limiting scalability and diversity of robotic skill data.
Approach
Use foundation models only for what they know (object semantics, affordances, task proposals, scene layouts, reward code, sizes), and let physics-based simulation with an automatically selected learning algorithm (RL, motion planning primitives, or gradient-based trajectory optimization) acquire the skills, enabling an endless generative propose-generate-learn cycle.
Key results
- RoboGen achieves the lowest Self-BLEU (0.284) and lowest embedding similarity in task descriptions (0.165) and images (ViT 0.193, CLIP 0.762) among compared benchmarks on 106 generated tasks.
- Average skill learning success rate of 0.774 over 69 benchmarked tasks (50 object manipulation, 7 soft-body, 12 locomotion).
- Manual review of 155 generated object manipulation tasks found 13 scene-generation failures and 6 training-supervision failures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RoboGen task generation instead of Behavior-100 | RoboGen · Task descriptions | Self-BLEU: 0.299 -> 0.284 | Helped |
| Use RoboGen task generation instead of ManiSkill2 | RoboGen · Task descriptions | SentenceBert embedding similarity: 0.194 -> 0.165 | Helped |
| Use RoboGen scene generation instead of ManiSkill2 | RoboGen · Generated scene images | ViT embedding similarity: 0.332 -> 0.193 | Helped |
| Use RoboGen scene generation instead of Behavior-100 | RoboGen · Generated scene images | CLIP embedding similarity: 0.833 -> 0.762 | Helped |
| Add VLM-based object verification (ablation) | RoboGen scene generation · 7 generated tasks | BLIP-2 score | Helped |
| Add plausible asset-size verification (ablation) | RoboGen scene generation · 7 generated tasks | BLIP-2 score | Helped |
| Add motion-planning primitives alongside RL (ablation) | RoboGen skill learning · 12 articulated-object manipulation tasks | success rate | Helped |
Surprising findings
- RL-only ablation fails completely on most articulated-object manipulation tasks, showing that algorithm selection is essential.
Limitations
- Large-scale verification of learned skills remains challenging, relying on limited checks
- Sim-to-real gap constrains real-world deployment
- LLM-generated scene and reward errors (e.g., misreading articulated joint states) cause failures
Experimental setup
Datasets: Objaverse PartNetMobility RLBench
Benchmarks: Behavior-100 RLBench MetaWorld Maniskill2 GenSim
Metrics: Self-BLEU sentence embedding similarity ViT image embedding similarity CLIP image embedding similarity BLIP-2 score skill learning success rate
Baselines: Behavior-100 RLBench MetaWorld Maniskill2 GenSim RL-only ablation w/o object verification w/o size verification
Builds on: Generative Simulation SAC BIT* Objaverse PartNetMobility
Closest papers by meaning
- ReGen: Generative Robot Simulation via Inverse Design (ICLR 2025 · similarity 0.88)
- Generative Control as Optimization: Time Unconditional Flow Matching for Adaptive and Robust Robotic Control (NeurIPS 2026 · similarity 0.88)
- GenSim: Generating Robotic Simulation Tasks via Large Language Models (ICLR 2024 · similarity 0.87)
- FactorSim: Generative Simulation via Factorized Representation (NeurIPS 2024 · similarity 0.87)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
- Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.