Research map / Label-efficient and robust learning / Domain adaptation and generalization
Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an unbalanced optimal transport loss with temporally aligned sampling for sim-and-real policy co-training, aligning joint observation-proprioception distributions so that abundant MimicGen simulation data improves real-world manipulation and generalizes to target-OOD states seen only in simulation.
Problem
Real-world behavior cloning needs costly demonstrations; simulation data can help but sim-to-real visual gaps and imbalance between abundant sim and scarce real demos hinder transfer and generalization to states covered only in simulation.
Approach
Align joint distributions of visual features and proprioceptive states across sim and real with an entropic unbalanced OT loss (KL marginal relaxation) added to BC co-training, plus a DTW-based trajectory-pair sampler that makes mini-batches contain behaviorally similar sim/real pairs.
Key results
- Real-world in-distribution average success: 0.73 (image) and 0.77 (point cloud) for the proposed method, versus 0.53 and 0.67 for co-training (image and point cloud respectively).
- Real-world OOD image-based average success rises to 0.40 for the proposed method versus 0.10 for co-training.
- Sim-to-sim image-based average target-OOD success is 0.36 for the proposed method versus 0.28 for co-training; point cloud average OOD is 0.59 vs 0.47.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add UOT-based sim-and-real co-training with temporally aligned sampling | Diffusion Policy, ResNet-18 · Sim-to-sim image, Target-OOD average | success rate: 0.28 -> 0.36 | Helped |
| add UOT-based sim-and-real co-training with temporally aligned sampling | 3D Diffusion Policy, PointNet · Sim-to-sim point cloud, Target-OOD average | success rate: 0.47 -> 0.59 | Helped |
| add UOT-based sim-and-real co-training | real-world image policy · Real-world image OOD, average full task | success rate: 0.10 -> 0.40 | Helped |
| add UOT-based sim-and-real co-training | real-world point-cloud policy · Real-world point cloud OOD, average full task | success rate: 0.50 -> 0.67 | Helped |
| add temporally aligned trajectory sampling (ablation) | UOT policy on Stack · Viewpoint1-Point, Target-OOD | success rate | Helped |
| use perfectly paired source-target state batches (ablation) | UOT policy on Stack · Viewpoint1-Point, Target-OOD | success rate | Helped |
| increase simulation trajectories from 100 to 1000 | Ours, Stack point-cloud policy · Real-world OOD regions without real-world coverage | success rate | Helped |
Surprising findings
- Target-only policies fail completely on OOD reset and texture conditions (0.00 OOD success in sim-to-sim), showing real-only data does not generalize.
Limitations
- Only addresses visual observation gaps; action dynamics gaps left for future work.
- Inherits MimicGen limitations: applies mainly to prehensile, quasi-static tasks.
- Requires a small set of real demonstrations aligned with simulated demonstrations.
Experimental setup
Datasets: MimicGen-generated simulation demonstrations (Robosuite) real-world teleoperated demonstrations (Franka Panda)
Benchmarks: Robosuite Lift Robosuite BoxInBin Robosuite Stack Robosuite Square Robosuite MugHang Robosuite Drawer real-world Franka tasks
Metrics: success rate
Baselines: MMD Co-training Source-only Target-only
Builds on: DeepJDOT Unbalanced optimal transport Diffusion Policy 3D Diffusion Policy MimicGen
Closest papers by meaning
- Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning (ICLR 2025 · similarity 0.89)
- In-Context Policy Adaptation via Cross-Domain Skill Diffusion (AAAI 2025 · similarity 0.88)
- DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control (ICML 2025 · similarity 0.88)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
- UniDomain: Pretraining a Unified PDDL Domain from Real-World Demonstrations for Generalizable Robot Task Planning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.