Research map / Large language models / Agents, code and math
HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumanoidGen, an LLM-driven framework that auto-generates bimanual dexterous manipulation tasks and demonstrations for a humanoid in simulation, using spatial annotations, constraint code, and MCTS reasoning.
Problem
Bimanual dexterous manipulation for humanoid robots lacks simulation tasks and high-quality demonstrations, and teleoperation or RL-based collection is costly and hard to scale.
Approach
An LLM planner decomposes long-horizon tasks into hand atomic operations and arm constraints (point/axis geometric relations on annotated keypoints and axes), solved by a trajectory optimizer; a Segment-Truncate-Combine-Resume MCTS variant with intrinsic "valuable moment" rewards improves reasoning under insufficient annotation; dynamic object-ignoring lists handle in-contact manipulation.
Key results
- MCTS raised reasoning success on block stack single by 55% with a 26% increase in token consumption (average tokens per successful execution fell from 24.17K to 19.63K).
- HumanoidGen averaged over 50% success across 20 tasks in generation, with most task types above 75%, outperforming RoboTwin on long-horizon and collision-heavy tasks.
- DP on open box hard rose from 11.1% success with 20 demos to 100% with 100 demos, surpassing DP3 (95.6%-98.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MCTS to LLM planning (ablation) | HumanoidGen task planning · Block Stack Single | reasoning success rate: 55% improvement | Helped |
| Add MCTS to LLM planning (ablation) | HumanoidGen task planning · Block Stack Single | tokens per successful execution method: 24.17K to 19.63K | Helped |
| Use MCTS instead of non-MCTS planning (ablation) | HumanoidGen plan generation · 20 successful plans | distinct plans generated: MCTS: ≥ 12; non-MCTS: ≤ 5 | Helped |
| Increase DP demonstrations from 20 to 100 | DP · Open Box Hard | success rate: 11.1% to 100.0% | Helped |
| Increase DP demonstrations from 20 to 100 | DP · Open Drawer | success rate: 17.8 to 48.9 | Helped |
| Increase DP demonstrations from 20 to 100 | DP · Open Laptop Hard | success rate: 15.6 to 35.6 | Helped |
| Increase DP3 demonstrations from 20 to 100 | DP3 · Dual Bottles Pick Hard | success rate: 88.9 to 94.4 | Helped |
| Increase DP3 demonstrations from 20 to 100 | DP3 · Open Drawer | success rate: 58.3 to 84.4 | Helped |
| Increase DP3 demonstrations from 20 to 100 | DP3 · Blocks Stack Easy | success rate: 0.0 to 22.8 | Helped |
| Increase demonstrations from 20 to 100 | DP3 · Cup Pour Easy | success rate: 67.8 to 72.2 | Mixed |
| Use DP3 instead of DP | policy trained on HumanoidGen data · Open Laptop Hard | success rate: DP: 35.6; DP3: 100.0 | Helped |
| Use DP3 instead of DP | policy trained on HumanoidGen data · Cup Pour Easy | success rate: DP: 0.0; DP3: 72.2 | Helped |
Surprising findings
- DP3 achieved 0% on blocks stack hard, handover and storage, and pyramid stack even with 100 demonstrations, indicating long-horizon tasks remain unlearned.
- Some DP results on close box hard were high with only 20 demos (82.2%) due to random arm movements, which the authors flag as unstable.
Limitations
- Long-horizon and fine-grained tasks remain unlearned by DP and DP3 even with 100 demonstrations.
- Relies on manual spatial annotation of assets and hands, with automatic annotation only partially evaluated.
- Evaluation of the policy only on the authors' own generated benchmark in simulation, with real-world experiments deferred to the appendix.
Experimental setup
Datasets: HGen-Bench tasks generated by HumanoidGen RoboCasa assets
Benchmarks: HGen-Bench
Metrics: success rate token consumption reasoning success rate number of distinct plans
Baselines: RoboTwin non-MCTS LLM planning
Builds on: RoboTwin Diffusion Policy 3D Diffusion Policy ManiSkill3 RoboGen
Closest papers by meaning
- ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.89)
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025 · similarity 0.87)
- RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills (NeurIPS 2025 · similarity 0.87)
- GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation (CVPR 2025 · similarity 0.87)
- BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects (CVPR 2025 · similarity 0.86)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.