Research map / Reinforcement learning / Imitation learning
Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DARAIL, which transfers a DARC policy learned in the source domain to the target domain via GAIfO-style state-only imitation with a doubly-robust-style reward augmented estimator, evaluated on MuJoCo off-dynamics tasks.
Problem
Off-dynamics RL: policies trained in a source domain with modified rewards (DARC) degrade when deployed in the target domain, because matching source trajectories to target-optimal ones does not guarantee target performance.
Approach
Use DARC's source-domain trajectories as expert demonstrations for state-only generative adversarial imitation from observation, with importance-weighted discriminator training and a doubly-robust-style reward augmented estimator R_AE that combines the discriminator reward with the true source reward.
Key results
- In the broken-source setting, DARAIL reaches 7067 on HalfCheetah versus 4133 for DARC evaluation in the target, and 5357 on Ant versus 4280.
- Under 1.5 gravity, DARAIL achieves 4409 on Walker2d versus 257 for DARC evaluation and 630 for GARAT.
- DARAIL outperforms IS-ACL on Ant and Walker2d in the broken-source setting and outperforms all baselines in the gravity setting, while being less sensitive to importance-weight clipping than IS-ACL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · HalfCheetah, broken source | return: 6402\pm362 -> 7067\pm176 | Helped |
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · Ant, broken source | return: 3239\pm395 -> 5357\pm79 | Helped |
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · Walker2d, broken source | return: 2330\pm156 -> 4366\pm434 | Helped |
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · Reacher, broken source | return: -13.9\pm1.1 -> -13.7\pm0.9 | Helped |
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · HalfCheetah, 1.5 gravity | return: 2666\pm 2037 -> 4093\pm 1021 | Helped |
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · Ant, 1.5 gravity | return: 990\pm 251 -> 3472\pm 771 | Helped |
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · Walker2d, 1.5 gravity | return: 525 \pm142 -> 4409\pm 401 | Helped |
| add reward augmentation to DAIL's imitation-learning reward estimator (ablation) | DARAIL · Reacher, 1.5 gravity | return: -16.5\pm1.1 -> -9.5\pm0.22 | Helped |
Surprising findings
- DARC's target-domain evaluation reward is roughly 40% lower than its source training reward on average in the broken-source setting.
- DARAIL can exceed the DARC source-domain training reward on some tasks, even though it is trained by imitating DARC.
Limitations
- Theoretical bound relies on assumptions such as bounded importance weights and bounded discriminator class, and the bound is not shown to be tight.
- Requires first training a DARC policy, so the pipeline needs more target rollouts and training than DARC alone.
- Evaluated only on four MuJoCo environments with two kinds of dynamics shift.
Experimental setup
Datasets: MuJoCo HalfCheetah-v2 Ant-v2 Walker2d-v2 Reacher-v2
Benchmarks: MuJoCo HalfCheetah-v2 Ant-v2 Walker2d-v2 Reacher-v2 under broken source and modified gravity/density shifts
Metrics: episode return in target domain
Baselines: DARC DAIL IS-R IS-ACL MBPO MATL GARAT
Builds on: DARC GAIL GAIfO Soft Actor-Critic doubly robust off-policy evaluation
Closest papers by meaning
- Diffusion-Reward Adversarial Imitation Learning (NeurIPS 2024 · similarity 0.89)
- Cross-Domain Policy Adaptation by Capturing Representation Mismatch (ICML 2024 · similarity 0.89)
- EvIL: Evolution Strategies for Generalisable Imitation Learning (ICML 2024 · similarity 0.89)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.88)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.