Research map / Large language models / Agents, code and math
GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GENMANIP, an Isaac Sim tabletop simulation platform with LLM-generated task-oriented scene graphs and the 200-scenario GENMANIP-BENCH, used to benchmark modular foundation-model manipulation systems against end-to-end policies (GR-1, ACT).
Problem
Real-world manipulation policy generalization is hard to benchmark fairly; existing simulators lack diverse, controllable instruction and scenario variation for comparing modular foundation-model systems with end-to-end imitation policies.
Approach
A task-oriented scene graph (ToSG) representation, compatible with LLMs, that lets GPT-4 synthesize diverse tabletop tasks and controllable layouts, with goal-condition-based automatic evaluation, plus a unified modular manipulation system abstracted from mark-based visual prompting methods.
Key results
- CoPA with GPT-4.5 reaches 23.0% overall SR on GENMANIP-BENCH, the best modular result, while long-horizon tasks average about 9% SR across models.
- GR-1 success on pick-banana-place task rises from 33.0% at 100 episodes to 95.0% at 1K episodes; ACT rises from 40.0% to 72.5%.
- GR-1 drops to 0.0% SR on unseen instructions and unseen objects, though it reaches 78.2% SR under limited region randomization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4.5 with CoPA rather than other evaluated modular systems | GENMANIP-BENCH · Overall | SR: 23.00 | Helped |
| Increase GR-1 behavior-cloning data from 100 to 1K episodes (ablation) | GR-1 · Single-task pick banana and place it on the plate | SR: 33.0% -> 95.0% | Helped |
| Increase ACT behavior-cloning data from 100 to 1K episodes (ablation) | ACT · Single-task pick banana and place it on the plate | SR: 40.0% -> 72.5% | Helped |
| Expand randomization from target-object position to all object positions in the limited region (ablation) | GR-1 · Limited region randomization | SR: 78.2±9.5 -> 57.0±8.4 | Hurt |
| Expand target-object position randomization from the limited region to the full table (ablation) | GR-1 · Position randomization | SR: 78.2±9.5 -> 45.5±10.1 | Hurt |
| Transfer across scene graphs and layouts (ablation) | GR-1 · Full table randomization across 5 scenes | SR: 45.5±10.1 -> 43.5±26.1 | Hurt |
| Use unseen instructions (ablation) | GR-1 · Unseen instruction | SR: 0.0±1.2 | Hurt |
| Use unseen object instances (ablation) | GR-1 · Unseen objects (scenarios) | SR: 0.0±0.0 | Hurt |
| Increase the number of goal conditions from 1 to 3 (ablation) | Modular manipulation system · Long-horizon tasks | SR: 32.28 -> 0.00 | Hurt |
| Add same-category distractor objects (ablation) | Modular manipulation system · Object distractors | SR: 20.27 -> 18.60 -> 18.00 | Hurt |
| Replace SoM grasp prompting with CtoF SoM (ablation) | Modular manipulation system with PointToPoint motion planning · Modular system subset | SR: 35.0 -> 27.5 | Hurt |
| Replace PointToPoint with Mark-based Grid motion planning (ablation) | Modular manipulation system with SoM · Modular system subset | SR: 35.0 -> 15.0 | Hurt |
Surprising findings
- Adding a single distractor object drops modular performance by 1.67 points, while further distractors reduce it only slightly, suggesting a threshold effect.
- Modular systems with CtoF SoM underperform plain SoM because repeated SoM calls compound errors with AnyGrasp proposals.
Limitations
- Full-scale evaluation across all 200 scenarios is left for future work due to computational constraints.
- End-to-end policies fail completely on unseen instructions and unseen objects.
- Primitive skill set is limited to pick-and-place and articulated skills; scaling diverse skill types is open.
Experimental setup
Datasets: Objaverse GRUtopia assets PartNet-Mobility
Benchmarks: GENMANIP-BENCH
Metrics: success rate SPL
Baselines: CoPA MOKA GR-1 ACT RoboCasa RLBench CALVIN LIBERO
Builds on: CoPA MOKA MimicGen Set-of-Mark prompting PDDL
Closest papers by meaning
- GenSim: Generating Robotic Simulation Tasks via Large Language Models (ICLR 2024 · similarity 0.89)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.87)
- DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control (ICML 2025 · similarity 0.87)
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.87)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.87)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.