Research map / Vision-language and multimodal / Robot action and navigation
Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes manipulation centricity, a Grad-CAM/SAM2-based metric correlated with downstream IL success, and MCR, a ResNet-50 representation pre-trained on the DROID robot dataset with dynamics alignment, action prediction, and temporal contrastive losses.
Problem
Pre-trained visual representations for robot manipulation are usually learned from human videos, which suffer from embodiment distribution shift and lack robot dynamics information; it is unclear which representation properties drive downstream manipulation performance.
Approach
Defines "manipulation centricity" (Jaccard overlap between Grad-CAM maps and SAM 2 ground-truth masks of end-effector and task objects), shows it correlates strongly with IL success (R=0.93), and pre-trains a ResNet-50 on DROID using proprioceptive state-action dynamics alignment, action prediction, and time-contrastive losses to raise it.
Key results
- MCR outperforms the strongest baseline by 14.8% across 20 simulation tasks in four domains (abstract).
- Manipulation centricity correlates with downstream IL success with Pearson R=0.93.
- MCR improves real-world success on three UR5e tasks by 76.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamics alignment loss (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 66.2\ci{0.8} -> 83.2\ci{1.3} | Helped |
| add action prediction loss (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 71.3\ci{1.2} -> 83.2\ci{1.3} | Helped |
| add temporal contrastive loss (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 72.0\ci{1.2} -> 83.2\ci{1.3} | Helped |
| change dynamic chunk length from 3 to 1 (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 83.2\ci{1.3} -> 72.1\ci{2.9} | Hurt |
| change dynamic chunk length from 3 to 5 (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 83.2\ci{1.3} -> 76.8\ci{2.4} | Hurt |
| change dynamic chunk length from 3 to 7 (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 83.2\ci{1.3} -> 76.8\ci{2.2} | Hurt |
| replace ResNet-50 with ResNet-18 (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 83.2\ci{1.3} -> 77.3\ci{1.8} | Hurt |
| replace ResNet-50 with ResNet-34 (ablation) | MCR · Can, Stick Pull, Laptop | success rate: 83.2\ci{1.3} -> 77.9\ci{1.7} | Hurt |
| pre-train R3M on DROID robot data rather than human data | R3M · gripper-based simulation tasks | downstream success rate | Helped |
| pre-train R3M on DROID robot data rather than human data | R3M · dexterous hand-based simulation tasks | downstream success rate | Hurt |
| use more DROID pre-training data | MCR · simulation downstream tasks | downstream performance | Helped |
| use MCR rather than baseline representations | robot policies · three real-world manipulation tasks | success rate: 76.9% | Helped |
Surprising findings
- R3M-DROID underperforms R3M on dexterous-hand tasks, suggesting a gripper-to-hand embodiment gap in DROID.
- Larger pre-training datasets help MCR, contrary to prior findings that scaling data does not necessarily help.
Limitations
- Dexterous-hand tasks show weaker gains because DROID contains only gripper data.
- Manipulation centricity relies on SAM 2 masks with manual point annotation and Grad-CAM, which adds evaluation cost.
Experimental setup
Datasets: DROID Robomimic RoboCasa MetaWorld DexArt
Benchmarks: Robomimic RoboCasa MetaWorld DexArt
Metrics: success rate Jaccard index Pearson correlation
Baselines: MVP VC-1 HRP R3M R3M-DROID LfS
Builds on: R3M MVP VC-1 HRP InfoNCE
Closest papers by meaning
- Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation (CVPR 2025 · similarity 0.92)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.92)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 · similarity 0.91)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.91)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.91)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.