Research map / Vision-language and multimodal / Robot action and navigation
RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RobotSmith, a pipeline where two VLM agents iteratively design parameterized tools, a VLM planner writes manipulation code with high-level APIs, and CMA-ES jointly optimizes tool geometry and trajectory in the Genesis simulator.
Problem
Robots need task-specific tools for complex manipulation, but retrieving human-designed tools yields objects hard for robot arms to use, and existing tool design methods are limited to fixed templates or generic 3D generation not optimized for function.
Approach
Combines a VLM proposer-critic design loop that outputs a modular, parameterized JSON/CSG-like tool representation (with optional text-to-3D parts) with a VLM-written trajectory built from minimal grasp/move/release APIs, then jointly optimizes tool shape and trajectory parameters with CMA-ES using physics-simulation task scores.
Key results
- Overall success rate 50.0% with P_best 0.94, versus 21.4% for Meshy 3D generation, 11.1% for retrieval, and 2.8% for no tool.
- Removing the trajectory optimizer drops success from 50% to 5% (best performance 0.28), the largest ablation drop.
- Real-world experiments with 3D-printed tools on an XArm7 for hold-a-phone and dough calabash tasks, plus a multi-step sesame pancake task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RobotSmith instead of Meshy | tool design and use pipeline · Overall | Success rate: 21.4% -> 50.0% | Helped |
| use RobotSmith instead of tool retrieval | tool design and use pipeline · Overall | Success rate: 11.1% -> 50.0% | Helped |
| use RobotSmith instead of no-tool planning | tool design and use pipeline · Overall | Success rate: 2.8% -> 50.0% | Helped |
| replace generated components with primitives only (ablation) | RobotSmith · Overall | Best performance: 0.94 -> 0.69 | Hurt |
| replace generated components with primitives only (ablation) | RobotSmith · Overall | Success rate: 50% -> 33% | Hurt |
| skip trajectory refinement and use the initial action plan (ablation) | RobotSmith · Overall | Best performance: 0.94 -> 0.28 | Hurt |
| skip trajectory refinement and use the initial action plan (ablation) | RobotSmith · Overall | Success rate: 50% -> 5% | Hurt |
| keep initial tool geometry fixed (ablation) | RobotSmith · Overall | Best performance: 0.94 -> 0.67 | Hurt |
| keep initial tool geometry fixed (ablation) | RobotSmith · Overall | Success rate: 50% -> 32% | Hurt |
Surprising findings
- Removing the trajectory optimizer causes the largest performance drop, showing VLM-generated trajectories alone are insufficient for precise execution.
- Meshy text-to-3D tools perform strongly on some tasks (hold a phone 67.5% best-of-8 success) despite over-emphasizing visual realism over function.
Limitations
- Optimization only covers scaling, rotation, and translation; no topology or part reconfiguration.
- Failures from Meshy outputs being uncontrollable in size/orientation, ambiguous tool orientation, and grasp API limits on heavy tools or abrupt motion.
- Evaluation uses 9 simulated tasks with 8 trials each, so success rates are coarse.
Experimental setup
Datasets: BlenderKit asset database
Benchmarks: Reach Hold a Phone Lift a Bowl Lift a Piggy Dough Calabash Flatten Dough Cut Dough Fill a Bottle Transport Water
Metrics: best performance P_best success rate over 8 trials
Baselines: No Tool Retrieval (BlenderKit) Meshy text-to-3D ShapeTalk 3D editing
Builds on: CMA-ES Constructive Solid Geometry Genesis simulator
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.87)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.87)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.87)
- GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation (CVPR 2025 · similarity 0.87)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.87)
- HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.