Research map / Reinforcement learning / Visual and representation RL
DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents DexScale, a data engine that projects real-world scenes and human videos into simulation and automates domain randomization and adaptation to scale simulated robot skill data for zero-shot Sim2Real policy transfer.
Problem
Simulated robot data is cheap and scalable, but policies trained on it often fail to transfer to real robots due to semantic mismatch between simulated and real scenes and simulation discrepancies (visual and physical).
Approach
A data engine that projects heterogeneous real-world priors (RGB scenes, human egocentric videos) into simulation, then automatically selects and samples domain randomization configurations (action-invariant and semantic-aware DR, with LLM-based feature ranking and ADR) plus object-oriented and pose-affordance domain adaptation to produce a scaled Sim2Real skill dataset for imitation policies.
Key results
- Across 10 Sim2Real gaps, DexScale reaches e.g. 80/100 sim and 5/10 real for camera position, versus 72/100 and 2/10 for skills-only data.
- For object pose and object shape gaps, DexScale improves sim success from 15/100 to 69/100 and 11/100 to 61/100 respectively versus skills-only.
- Handcrafted DR pick-and-place reaches 0.79 sim / 0.56 real, while the text reports a rising trend with automatic DR, though the DexScale value for that setting is not directly comparable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add domain randomization and domain adaptation to skill-only training (ablation) | DexScale · Real deployment across the ten Table 2 domain gaps | success rate: 0/10 -> 4/10; 1/10 -> 6/10; 2/10 -> 4/10; 1/10 -> 3/10; 0/10 -> 4/10; 2/10 -> 5/10; 0/10 -> 3/10; 1/10 -> 6/10; 1/10 -> | Helped |
| Add domain randomization (ablation) | Skills Only · Real deployment across the ten Table 2 domain gaps | success rate | Helped |
| Add domain adaptation (ablation) | Skills Only · Real deployment across the ten Table 2 domain gaps | success rate | Helped |
Surprising findings
- Policies trained on skills-only data reach near-zero real-world success (0-2 out of 10) across most gaps, showing how severe the Sim2Real gap is without scaling.
- The automatic DR discovered that randomizing textures and backgrounds helps bridge distractor-related gaps, though textures are not directly related to distractors.
Limitations
- Real-world evaluation uses only 10 trials per setting, so estimates are noisy.
- Failure cases include unobservable grasp orientation, incorrect grasp pose and depth, and cross-chunk jitter.
- Scope limited to simple tasks and arms; extension to humanoids and long-horizon tasks is future work.
Experimental setup
Datasets: Objaverse-XL egocentric human videos real-world RGB scenes
Benchmarks: simulated and real-world grasping success rates across 10 Sim2Real domain gaps real-world box opening, table rearrangement, water pouring
Metrics: success rate
Baselines: Skills Only dataset Skills+DR Skills+DA handcrafted DR
Builds on: ADR (automatic domain randomization) Digital Cousin Diffusion Policy ACT (Action Chunking with Transformers) RDT
Closest papers by meaning
- DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation (NeurIPS 2025 · similarity 0.89)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.88)
- GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation (CVPR 2025 · similarity 0.87)
- Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning (ICLR 2025 · similarity 0.87)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.