Research map / Reinforcement learning / Visual and representation RL
What Matters in Learning from Large-Scale Datasets for Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses a procedural simulation data generator (MimicLabs, built on BDDL, LIBERO and MimicGen) to study which dataset diversity dimensions matter for collecting and retrieving robot demonstrations, validated with real-robot experiments on DROID retrieval.
Problem
Lack of a systematic understanding of what data should be collected for large-scale robot manipulation datasets and how practitioners should retrieve relevant demonstrations from existing datasets such as DROID for downstream policy learning.
Approach
Procedurally generate robot datasets with controlled dimensions of variation (camera pose, object/table texture, spatial arrangements, motion) using BDDL task specs and MimicGen demo synthesis, then run controlled collector and retriever co-training studies that isolate which DVs need alignment versus diversity.
Key results
- In simulation, co-training with diverse camera poses boosted clear-table success from ~43% to 90% over baseline when camPose was misaligned with the target.
- Retrieving skill-relevant demos plus camera pose alignment gave 96.67% on bin carrot versus 50% target-only in MimicLabs.
- On real DROID retrieval, aligned retrieval outperformed target-only and full-DROID co-training, with boosts of 5% to 80%; full-DROID co-training failed to learn the tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase co-training camera-pose diversity (ablation) | BC-RNN on clear table with misaligned camera poses · clear table | success rate: 43.33 -> 90 | Helped |
| Increase co-training object-texture diversity (ablation) | BC-RNN on clear table with misaligned object textures · clear table | success rate: 93.33 -> 90 | No clear effect |
| Increase co-training table-texture diversity (ablation) | BC-RNN on clear table with misaligned table textures · clear table | success rate: 66.67 -> 63.33 | No clear effect |
| Increase co-training object-spatial diversity (ablation) | BC-RNN on clear table with misaligned object spatial arrangements · clear table | success rate: 26.67 -> 56.67 | Helped |
| Align retrieved camera poses with the target (ablation) | BC-RNN · microwave mug | success rate: 46.67 -> 70 | Helped |
| Align retrieved camera poses with the target (ablation) | BC-RNN · clear table | success rate: 20 -> 36.67 | Helped |
| Retrieve aligned object textures (ablation) | BC-RNN · microwave mug | success rate: 66.67, 73.33, 70 | No clear effect |
| Retrieve relevant object/skill demonstrations and align camera poses (ablation) | BC-RNN · bin carrot | success rate: 50 -> 96.67 | Helped |
| Retrieve relevant object/skill demonstrations and align all listed dimensions (ablation) | BC-RNN · bin bowl | success rate: 33.33 -> 73.33 | Helped |
| Retrieve relevant object/skill demonstrations (ablation) | BC-RNN · clear table (10 target demos) | success rate: 23.33 -> 20 | Hurt |
| Retrieve relevant object/skill demonstrations and align camera poses (ablation) | BC-RNN · clear table (20 target demos) | success rate: 36.67 -> 46.67 | Helped |
| Retrieve relevant object/skill demonstrations and align camera poses (ablation) | BC-RNN · microwave teapot (20 target demos) | success rate: 30 -> 50 | Helped |
Surprising findings
- Object texture misalignment had minimal impact on downstream performance in both collection and retrieval settings.
- Co-training on all of DROID failed to learn the real-world tasks, while small aligned retrievals performed better.
Limitations
- Main controlled studies are in simulation with a limited set of tasks and DVs.
- Real-world experiments cover only seven tasks and a single robot setup.
- DROID retrieval required building custom metadata (object labels, colors) with a VLM and heuristics.
Experimental setup
Datasets: MimicLabs DROID
Benchmarks: MimicLabs clear table microwave mug bin carrot bin bowl microwave teapot make coffee
Metrics: task success rate
Baselines: Target only Full DROID co-training BC-RNN BC-Transformer
Builds on: MimicGen LIBERO BDDL Diffusion Policy DROID
Closest papers by meaning
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.91)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 · similarity 0.88)
- STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning (ICLR 2025 · similarity 0.87)
- GraphMimic: Graph-to-Graphs Generative Modeling from Videos for Policy Learning (CVPR 2025 · similarity 0.87)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.