Research map / Image generation / Autoregressive and latent generation
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RoboTwin, a generative digital-twin framework that builds 3D object assets from single RGB images and uses LLM-driven spatial-annotation code generation to produce dual-arm expert demonstrations and a sim+real benchmark.
Problem
Dual-arm robot manipulation lacks diverse high-quality demonstration data and real-world-aligned evaluation benchmarks, and existing simulated data generators are tied to fixed scenarios or single-arm tasks.
Approach
Generate diverse digital-twin object assets from one 2D image using a 3D generative model, annotate functional/approach/lateral axes and contact points on assets (transferred across instances via diffusion feature matching), and have an LLM decompose tasks and write constraint-based motion code to produce expert demonstrations.
Key results
- Pre-training on 300 simulated samples and fine-tuning on 20 real samples raised real-world single-arm average success from 1.2% to 72%.
- Dual-arm real-world average success rose from 20% (20 real) to 62% (300 sim + 20 real).
- DP with 100 demonstrations reached 85.7% on Dual Bottles Pick (Easy), while DP3 was stronger with few demos; Dual Shoes Place stayed below 15% across methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on 300 simulation datasets, then fine-tune on 20 real-world datasets | Real-world single-arm policies · Single-arm real-world tasks | success rate: 1.2% -> 72% | Helped |
| Pre-train on 300 simulation datasets, then fine-tune on 20 real-world datasets | Real-world dual-arm policies · Dual-arm real-world tasks | success rate: 20% -> 62% | Helped |
| Increase expert demonstrations from 20 to 100 | Diffusion Policy (DP) · Dual Bottles Pick (Easy) | success rate: 1.7% -> 85.7% | Helped |
| Use XYZ+RGB point-cloud input instead of XYZ-only input | DP3 · RoboTwin benchmark tasks | success rate | Mixed |
Surprising findings
- DP3 with RGB added gives inconsistent benefits, improving cluttered Pick Apple Messy but degrading Container Place.
- 2D Diffusion Policy scores 0% with 20 demonstrations on most tasks but scales strongly with data, outperforming DP3 at 100 demos on several tasks.
Limitations
- Dual-arm coordination tasks such as Dual Shoes Place remain poorly solved by current imitation learning methods.
- Diverse initial bottle states (upright or lying down) make dual-arm tasks substantially harder in real-world tests.
- Asset quality checks rely on GPT-4V and CLIP-based metrics, which may not fully ensure physical fidelity.
Experimental setup
Datasets: RoboTwin simulated expert data (generated) real-world teleoperated data on COBOT Magic platform
Benchmarks: RoboTwin (14-15 dual-arm tasks) real-world single-arm and dual-arm tasks on COBOT Magic
Metrics: success rate
Baselines: Diffusion Policy (2D) 3D Diffusion Policy (DP3) DP3 (XYZ+RGB)
Builds on: MimicGen Code as Policies ReKep Diffusion Policy 3D Diffusion Policy
Closest papers by meaning
- RoboPARA: Dual-Arm Robot Planning with Parallel Allocation and Recomposition Across Tasks (ICLR 2026 · similarity 0.88)
- Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination (ICLR 2025 · similarity 0.86)
- SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation (ICML 2025 · similarity 0.86)
- RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills (NeurIPS 2025 · similarity 0.86)
- HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning (NeurIPS 2025 · similarity 0.85)
- ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.