Research map / Vision-language and multimodal / Robot action and navigation

Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation

CVPR 2025 · Vision-language and multimodal › Robot action and navigation · approach: Memory and subgoal-based manipulation · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts human-data pre-trained visual encoders (R3M, D4R) to the robot domain with parameter-efficient adapters trained via a human-robot contrastive alignment loss on paired RH20T demonstrations, improving manipulation success on Adroit, RLBench, and real-world tasks.

Problem

Visual representations pre-trained on human activity videos transfer poorly to robotic manipulation because of the morphological human-robot domain discrepancy, and existing fixes either lack robot data or require per-environment customization.

Approach

Use paired human-robot demonstrations as bridges: insert lightweight adapters into a frozen human-pretrained encoder and train them on robot videos with a contrastive loss that aligns adapted robot features with frozen human features (and task-aware text-queried attention pooling), keeping the backbone versatile.

Key results

• On RLBench 18 multi-tasks, R3M-Align improves average success rate by 8.9% (50.3 to 59.2) and D4R-Align by 4.6% (55.3 to 59.9).
• On Adroit two tasks, R3M-Align averages 81.3 vs 74.0 for R3M (+7.3%) while learning only 1.6M parameters versus 25M backbone.
• On five real-world tasks, D4R-Align shows a 13% average success improvement over D4R and R3M-Align an 11% improvement over R3M.

Measured effects reported in the paper

Surprising findings

• Inserting adapters at all three positions (E.M.L) did not outperform single-position insertion, suggesting difficulty harmonizing adaptation across multiple positions.
• Full fine-tuning baselines (R3M-PreT, R3M-ClS) gave smaller gains than the parameter-efficient alignment method on Adroit.

Limitations

• Does not measure or quantify the human-robot domain discrepancy directly.
• Adapts pre-trained models without revisiting pre-training or combining paired robot data with out-of-domain human data.
• Effect of scale and diversity of pre-training and paired data is not studied.

Experimental setup

Datasets: RH20T Ego4D Kinetics Adroit RLBench

Benchmarks: Adroit RLBench real-world xArm7 tasks

Metrics: success rate average success rate t-SNE feature distribution

Baselines: R3M D4R R3M-PreT R3M-ClS RVT

Builds on: R3M data4robotics RVT ACT

Closest papers by meaning

• Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.92)
• Solving New Tasks by Adapting Internet Video Knowledge (ICLR 2025 · similarity 0.89)
• Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.89)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation (CVPR 2025 · similarity 0.88)
• Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.88)
• Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.88)