Research map / Vision-language and multimodal / Robot action and navigation
CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects a small set of simulated teleoperated human demos, augments them extensively in SAPIEN (camera views, lighting/texture, diverse objects, object pose via sensitivity-aware kinematics), trains an ACT policy with automatic curriculum learning, and fine-tunes on 3 minutes of real demos for dexterous Allegro-hand manipulation.
Problem
Real-world imitation learning for dexterous multi-finger manipulation needs large amounts of costly real human demonstrations, and policies trained on limited real data are brittle to visual, geometric and pose variations and to sim2real gaps.
Approach
Use simulator-based trajectory-level data augmentation of simulated human teleoperation demos (including sensitivity-aware kinematics for object pose changes and physically rendered camera/light/texture changes), train with an automatic curriculum driven by policy success rate, and fine-tune on a few real demos to bridge sim2real.
Key results
- On real robot, Ours achieves 7/20 to 17/20 across tasks and levels vs R3M, PVR, MVP baselines, with an average 31.67% boost in in-domain Level 1.
- For rotating novel tetra-valve and penta-valve, CyberDemo achieves a 42.5% success rate versus at most 2.5% for baselines in the hardest setting.
- Ablation: training with all four augmentation levels (810 demos) reaches 92.5% in sim and 35-40% on real tests, outperforming 100 real-only-style settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Levels 1–4 simulator data augmentation instead of using Level 1 only (ablation) | CyberDemo · Real-world settings | success rate: 20% -> 35%; 5% -> 30%; 0% -> 30%; 0% -> 40% | Helped |
| add Levels 1–4 simulator data augmentation instead of using Level 1 only (ablation) | CyberDemo · Simulation, Level 1 | success rate: 78% -> 92.5% | Helped |
| add Levels 1–4 simulator data augmentation instead of using Level 1 only (ablation) | CyberDemo · Simulation, Level 2 | success rate: 0% -> 81% | Helped |
| base curriculum progression on task success rate rather than data generation rate (ablation) | CyberDemo · Real world, Out of Position + Random Light | success rate: 20% -> 35% | Helped |
| remove curriculum learning from auto-domain randomization (ablation) | CyberDemo · Real world, Out of Position + Random Light | success rate: 5% -> 35% | Hurt |
| replace real-demo-only training with simulated demonstrations, augmentation, and real-demo fine-tuning | real-world dexterous manipulation · Pick and Place, Bottle, In Domain | successes / trials: 4 / 20 -> 7 / 20 | Helped |
| replace real-demo-only training with simulated demonstrations, augmentation, and real-demo fine-tuning | real-world dexterous manipulation · Pick and Place, Can, In Domain | successes / trials: 7 / 20 -> 14 / 20 | Helped |
| replace real-demo-only training with simulated demonstrations, augmentation, and real-demo fine-tuning | real-world dexterous manipulation · Pouring, In Domain | successes / trials: 3 / 20 -> 9 / 20 | Helped |
| replace real-demo-only training with simulated demonstrations, augmentation, and real-demo fine-tuning | real-world dexterous manipulation · Rotating tri-valve, In Domain | successes / trials: 11 / 20 -> 15 / 20 | Helped |
| add training on diverse simulated objects | CyberDemo · Rotating unseen objects | success rate: 7.5 -> 42.5 | Helped |
| add training on diverse simulated objects | CyberDemo · Pick and Place, unseen objects | success rate: 25 -> 65 | Helped |
Surprising findings
- Adding more randomized simulated augmentation levels improved performance even on simpler in-domain simulation settings.
- Simulated augmented demos with fine-tuning outperformed in-domain real-world demonstrations on the real robot.
Limitations
- Requires designing a simulated environment for each real-world task, adding human effort.
- Sim2real controller gap remains for high-DoF dexterous hand, especially for non-quasi-static tasks.
Experimental setup
Datasets: simulated human teleoperation demonstrations (SAPIEN) real-world teleoperation demonstrations (3 min per task)
Benchmarks: real-robot pick and place real-robot pouring real-robot rotate (valve) SAPIEN simulation tests
Metrics: success rate
Baselines: R3M PVR MVP
Builds on: Action Chunking with Transformers MimicGen domain randomization
Closest papers by meaning
- DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation (NeurIPS 2025 · similarity 0.87)
- ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.86)
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.85)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.85)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.