Research map / Vision-language and multimodal / Robot action and navigation
Data Scaling Laws in Imitation Learning for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of data scaling in imitation learning showing policy generalization to unseen environments and objects follows roughly power laws in the number of environments/objects, with diversity mattering more than demonstration count, enabling zero-shot single-task policies at ~90% success using a Diffusion Policy with DINOv2 ViT-L/14 trained on UMI-collected data.
Problem
Whether data scaling laws exist in robotic manipulation imitation learning, and how many environments, objects, and demonstrations are needed for single-task policies to generalize zero-shot to new environments and objects within the same category.
Approach
Systematically scales the number of training environments, objects, and environment-object pairs with UMI hand-held gripper demonstrations, fits power laws to optimality gap, and derives a data collection recipe (one object per environment, ~32 environment-object pairs, 50 demonstrations each) that is verified on two new tasks.
Key results
- Object and environment generalization scores follow approximate power laws with number of training objects, environments, and environment-object pairs.
- With 32 environment-object pairs and 50 demonstrations each, policies achieve 0.922-0.95 normalized score and 85-92.5% success rate across four tasks in 8 unseen environments.
- Scaling the DINOv2 visual encoder from ViT-S/14 (0.66) to ViT-L/14 (0.90) improves Pour Water score; scaling the U-Net does not help.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale number of environment-object pairs (power-law fit) | Mouse Arrangement policy · unseen environments and objects | normalized score | Helped |
| Increase number of training objects from 8 to 32 objects | Pour Water and Mouse Arrangement policies · unseen objects, same environment | normalized score | Helped |
| Increase diversity of environments/objects vs. demonstrations per environment | Pour Water and Mouse Arrangement policies · unseen environments and objects | normalized score | Helped |
| Increase demonstrations per environment/object beyond a threshold (about 800 total for 16 pairs) | Pour Water and Mouse Arrangement policies · unseen environments and objects | normalized score | No clear effect |
| Use multiple objects per environment instead of one (at large environment counts) | Pour Water and Mouse Arrangement policies · unseen environments with 2 novel objects each | normalized score | No clear effect |
| Use learning from scratch instead of pre-trained visual encoder (ablation) | DINOv2 ViT-L/14 visual encoder · Pour Water | normalized score: 0.90 -> 0.03 | Hurt |
| Freeze DINOv2 visual encoder (no fine-tuning) (ablation) | DINOv2 visual encoder · Pour Water | normalized score: 0.90 -> 0.00 | Hurt |
| Replace full fine-tuning with LoRA (rank 8) on DINOv2 (ablation) | DINOv2 visual encoder · Pour Water | normalized score: 0.90 -> 0.72 | Hurt |
| Scale visual encoder from ViT-S/14 to ViT-L/14 (ablation) | DINOv2 visual encoder · Pour Water | normalized score: 0.66 -> 0.81 -> 0.90 | Helped |
| Scale action diffusion U-Net from base to large (ablation) | Diffusion Policy action head · Pour Water | normalized score: 0.90 -> 0.83 | Hurt |
| Use MSE on validation set as evaluation metric instead of tester scores | evaluation metric · Pour Water object generalization | correlation with real-world score | Hurt |
Surprising findings
- Scaling the action diffusion U-Net does not improve performance and the largest U-Net slightly declines (0.90 to 0.83).
- Action MSE on validation set often does not correlate with real-world performance, e.g., LoRA has lower MSE but worse real scores.
Limitations
- Only single-task policies studied; no task-level or language-conditioned generalization.
- Only imitation learning with Diffusion Policy; RL and other policy algorithms not explored.
- Only four tasks evaluated; UMI SLAM introduces demonstration errors and fails in texture-poor environments.
Experimental setup
Datasets: UMI-collected human demonstrations across in-the-wild environments and objects
Benchmarks: Pour Water Mouse Arrangement Fold Towels Unplug Charger
Metrics: normalized score (tester-assigned) success rate action MSE
Baselines: ImageNet ResNet CLIP ViT LoRA DINOv2 frozen DINOv2 learning from scratch ViT-L/14
Builds on: Diffusion Policy UMI DINOv2 ACT
Closest papers by meaning
- What Matters in Learning from Large-Scale Datasets for Robot Manipulation (ICLR 2025 · similarity 0.91)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.90)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.89)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 · similarity 0.88)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.88)
- What's the Move? Hybrid Imitation Learning via Salient Points (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.