Research map / Large language models / Agents, code and math
GenSim: Generating Robotic Simulation Tasks via Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses GPT-4 prompting (goal-directed and exploratory modes) with a retrieval task library to generate over 100 Ravens-style simulation tasks with code and demonstrations, then trains CLIPort-style multitask policies and transfers them to real-world robots.
Problem
Simulation data for robot policies is limited in task-level diversity because creating and verifying novel simulation tasks requires substantial human effort, so policies trained in simulation generalize poorly across tasks.
Approach
Treat each robot simulation task as executable code and have an LLM propose, implement, validate (execution, LLM critic, policy training) and store tasks in a retrieval-augmented task library; the generated tasks then serve as demonstration data for multitask language-conditioned policy pretraining, followed by sim-to-real adaptation.
Key results
- Pretraining on 70 GPT-4-generated tasks yields 62.5% average real-world success over 12 long-horizon tasks, versus 39.2% for CLIPort-pretrained baseline and 27.5% without pretraining.
- Joint training with GPT-generated tasks improves few-shot performance on original CLIPort tasks by over 50%.
- GPT-4 generates over 100 tasks (120 exploratory tasks); GPT-generated task simulation success rates (about 75%) are similar to human-curated tasks (about 77%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain on 70 rather than 50 GPT4-generated simulation tasks (ablation) | CLIPort policy · Real-world 12-task evaluation | success rate: 46.7% -> 62.5% | Helped |
| pretrain on 70 GPT4-generated simulation tasks rather than CLIPort tasks | CLIPort policy · Real-world 12-task evaluation | success rate: 39.2% -> 62.5% | Helped |
| add GPT4-generated tasks to joint policy training | CLIPort policy · Original CLIPort tasks, especially 5-demo few-shot setting | policy performance: over 50% | Helped |
| pretrain on more LLM-generated tasks (ablation) | CLIPort policy · Unseen tasks in the original Ravens benchmark | zero-shot generalization | Helped |
| use a two-stage task-description/code prompt with few-shot examples and task-library references rather than a single prompt or zero-shot prompt (ablation) | Exploratory LLM task generation · Robotic simulation task generation | code-generation success rate | Helped |
| fine-tune GPT3.5 and Code Llama on GPT4-generated tasks (ablation) | LLM task creator · Robotic simulation task generation | code-generation capability | Helped |
| use simulation-pretrained weights rather than no pretraining (ablation) | Real-world policy · Real-world 12-task evaluation | success rate: 0% -> 62.5% | Helped |
| pretrain on GPT4-generated tasks rather than human-curated tasks (ablation) | Single-task and multi-task policies · Generated simulation tasks | task success rate: 75.8% vs. 76.6%; 74.1% vs. 76.1% | No clear effect |
| provide per-step instructions rather than a high-level task goal (ablation) | Real-world single-task policy · Real-world tasks | policy performance | No clear effect |
| train on GPT4-generated tasks rather than only the 10 CLIPort tasks (ablation) | Multi-task policy · GPT-generated tasks | success rate: lower than 5% | Helped |
Surprising findings
- Policies pretrained on all 10 CLIPort tasks achieve below 5% success on GPT-generated tasks.
- Code Llama could not produce sensible code with prompting alone and only worked after LoRA finetuning.
Limitations
- Generated code contains syntax errors, hallucinations, and lacks physical/geometric grounding; generated tasks often need manual filtering.
- Only top-down pick-and-place tabletop tasks are explored; dexterous or contact-rich tasks are not studied.
- Language instructions and task behaviors can be misaligned, which the evaluation metrics do not capture.
Experimental setup
Datasets: Ravens benchmark (10 human-curated tasks) GPT-4-generated task library
Benchmarks: Ravens benchmark / CLIPort tasks GenSim generated tasks real-world XArm-7 tasks
Metrics: syntax-correct pass rate runtime-verified pass rate task completed pass rate task success score 0-100 real-world success rate
Baselines: CLIPort GPT-3.5 GPT-4 Code Llama-Instruct 7B Code Llama-Instruct 13B No Pretrain No Adaptation
Builds on: CLIPort Ravens benchmark Code Llama LoRA Codex
Closest papers by meaning
- GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation (CVPR 2025 · similarity 0.89)
- RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation (ICML 2024 · similarity 0.87)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.87)
- GenPlan: Generative Sequence Models as Adaptive Planners (AAAI 2025 · similarity 0.87)
- GNN-Transformer Task Planning Enhanced with Semantic-Driven Data Augmentation (AAAI 2025 · similarity 0.87)
- ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.