Research map / Reinforcement learning / Visual and representation RL
Cross-Domain Policy Adaptation by Capturing Representation Mismatch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAR, which learns state and state-action encoders only on target-domain transitions and uses the resulting representation deviation on source transitions as a reward penalty for SAC-based cross-domain policy adaptation.
Problem
Policy adaptation under dynamics mismatch between a source domain with abundant data (online or offline) and a target domain with only limited online interactions, where state, action and reward spaces are shared.
Approach
Learn deterministic state and state-action encoders only from target-domain data with a latent dynamics consistency loss, then penalize source-domain rewards by the representation deviation of source transitions; theory links this deviation to the return gap between domains.
Key results
- PAR achieves the best performance on 6 of 8 online tasks and beats SAC-tune on 7 of 8.
- With offline source data, PAR surpasses baselines on 17 of 24 tasks.
- On halfcheetah (no thighs) with medium-expert data, PAR reaches 10517 vs. about 2740 for the strongest baseline (roughly 4x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PAR instead of SAC-tar | Online policy adaptation · 8 kinematic and morphology-shift tasks | target-domain return | Helped |
| Use PAR instead of the strongest baseline | Online policy adaptation · 8 kinematic and morphology-shift tasks | target-domain return: 6 out of 8 tasks | Mixed |
| Use PAR instead of SAC-tune | Online policy adaptation · 8 kinematic and morphology-shift tasks | target-domain return: 7 out of 8 tasks | Helped |
| Use PAR with offline source data instead of baseline methods | Offline policy adaptation · 24 tasks | target-domain return: 17 out of 24 tasks | Helped |
| Remove the representation-mismatch reward penalty (set β=0) (ablation) | PAR · halfcheetah (broken back thigh) and walker (no right thigh) | final return | Hurt |
| Use a large reward penalty coefficient β (ablation) | PAR · some tasks, including walker | return | Hurt |
| Replace the representation-learning objective with the PAR-B objective (ablation) | PAR · four kinematic and morphology-mismatch tasks | target-domain return | No clear effect |
| Increase target-domain interaction interval F (ablation) | PAR · online policy adaptation | return | Helped |
| Increase source dataset quality to medium-expert (ablation) | PAR · many offline tasks | target-domain return | Helped |
| Use medium-replay source datasets | PAR · some offline tasks | target-domain return | Hurt |
Surprising findings
- DARC's reward penalty approaches 0 on halfcheetah but grows large on walker, suggesting an unstable penalty signal.
- PAR-B (a variant where the encoder receives raw state and action) performs nearly identically to PAR.
Limitations
- The penalty coefficient beta must be tuned manually per task.
- Performance is less satisfying on some medium-replay offline datasets with large diversity.
- Theory relies on a one-to-one state-action to latent mapping assumption that is justified only heuristically.
Experimental setup
Datasets: OpenAI Gym halfcheetah hopper walker ant D4RL -v2 datasets
Benchmarks: OpenAI Gym halfcheetah, hopper, walker, ant with kinematic and morphology shifts D4RL-based offline source tasks (24 tasks)
Metrics: average return in target domain standard deviation over seeds
Baselines: SAC-tar DARC DARC-weight VGDF SAC-tune CQL-0 CQL+SAC H2O VGDF+BC
Builds on: SAC DARC VGDF CQL
Closest papers by meaning
- Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation (NeurIPS 2024 · similarity 0.89)
- In-Context Policy Adaptation via Cross-Domain Skill Diffusion (AAAI 2025 · similarity 0.88)
- Multi-Agent Domain Calibration with a Handful of Offline Data (NeurIPS 2024 · similarity 0.88)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.