Research map / Multi-agent systems and games / Human-AI and goal-conditioned
ReGen: Generative Robot Simulation via Inverse Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReGen, an inverse-design generative simulation framework that uses LLM-guided graph expansion of causal event/entity/property graphs, converted to FSM-constrained symbolic programs, to generate CARLA driving and PyBullet manipulation environments conditioned on robot behaviors.
Problem
Constructing diverse, simulatable robot environments is labor-intensive, and existing generative simulation methods struggle to generate low-level behaviors/trajectories from text and require training new policies for each generated reward-driven task.
Approach
Invert the usual pipeline: start from an observed agent behavior (trajectory or reward) and expand a causal directed graph backward via LLM node proposal and batch edge classification, then ground the resulting scenario as an FSM and CP-SAT-solved simulation config.
Key results
- Generated driving scenarios achieve 80% FSM feasibility success rate, and the method reports higher scenario diversity than ChatScene, DriveLM, and the CARLA Leaderboard baselines on embedding and Self-BLEU metrics.
- Manipulation generation reaches 78% success rate, yielding 38 environments from 10 RoboGen reward functions with the highest diversity among compared baselines.
- GPT-4o accuracy on the generated driving VLA dataset drops to 0.63 versus 0.87 on DriveCoT and 0.90 on DriveLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate driving scenarios with ReGen instead of DriveCoT or DriveLM | GPT-4o · ReGen driving dataset | accuracy (%): DriveCoT 0.87; DriveLM 0.90; ReGen 0.63 | Hurt |
| Generate driving scenarios with ReGen instead of DriveCoT or DriveLM | GPT-4-Turbo · ReGen driving dataset | accuracy (%): DriveCoT 0.80; DriveLM 0.83; ReGen 0.53 | Hurt |
| Generate driving scenarios with ReGen instead of DriveCoT or DriveLM | Claude 3.5 Sonnet · ReGen driving dataset | accuracy (%): DriveCoT 0.80; DriveLM 0.77; ReGen 0.50 | Hurt |
| Reuse RoboGen reward functions while changing the environmental context | ReGen manipulation simulations · Manipulation task diversity | diversity: 18.50% | Helped |
Surprising findings
- VLMs frequently default to deceleration/braking as the ego action, failing on lane-change scenarios such as overtaking or debris avoidance.
- Most failures stem from overly strict FSM constraints requiring multiple conditions to hold simultaneously, rather than semantic errors.
Limitations
- Properties not supported by the simulator (e.g., temperature, sound) cannot be simulated, limiting manipulation tasks.
- Articulated objects such as shelves and doors cannot have objects placed inside them, causing manipulation failures.
- Overly strict FSM constraints cause infeasible scenarios; the paper also notes manual conversion of scenarios to Scenic files for SafeBench.
Experimental setup
Datasets: CARLA Leaderboard 2.0 scenarios NuScenes DriveLM DriveCoT
Benchmarks: CARLA PyBullet SafeBench
Metrics: Self-BLEU embedding diversity (1 - Sentence-BERT similarity) success rate collision rate VLM action accuracy
Baselines: ChatScene DriveLM DriveCoT CARLA Leaderboard 2.0 RoboGen GenSim Behavior-100 RLBench Learn-to-Collide AdvSim Carla Scenario Generator Adversarial Trajectory Optimization
Builds on: ChatScene RoboGen Soft Actor-Critic
Closest papers by meaning
- Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion (NeurIPS 2024 · similarity 0.89)
- RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation (ICML 2024 · similarity 0.88)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
- FactorSim: Generative Simulation via Factorized Representation (NeurIPS 2024 · similarity 0.86)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.86)
- Instant Policy: In-Context Imitation Learning via Graph Diffusion (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.