Research map / Label-efficient and robust learning / Domain adaptation and generalization
Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IGDF, a contrastive representation-based mutual-information-gap estimator that filters source-domain transitions for cross-domain offline RL, used as an add-on to IQL.
Problem
Merging source and target offline data in cross-domain offline RL causes performance degradation due to dynamics mismatch; existing dynamics-gap measures (domain classifiers, dynamics models) are unbounded or unreliable under large shifts.
Approach
Measure the domain gap as a mutual-information gap between state-action pairs and next states, estimated with an InfoNCE-style contrastive objective (target transitions as positives, source next-states paired with target state-actions as negatives), and use learned score functions to keep top-quantile source transitions and reweight their TD errors.
Key results
- IGDF with IQL achieves the highest summed score over 18 body-mass/joint-noise tasks (639.05 and 654.85) using 10% target data, with the smallest degradation relative to 100% target baselines (-11.89% and -10.81%).
- IGDF attains state-of-the-art results on 17 of 18 broken-thigh and morphology tasks, with a sum of 538.31 (broken) and 525.71 (morphology) versus 488.61 and 430.83 for the strongest baselines.
- Abstract claims IGDF uses only 10% of target data to reach 89.2% of the performance of state-of-the-art methods trained on 100% target data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter source transitions using IGDF contrastive scores instead of training on mixed source and target data without filtering | IQL · Body mass shift (9 tasks) | sum of normalized scores: 351.21 -> 639.05 | Helped |
| Filter source transitions using IGDF contrastive scores instead of training on mixed source and target data without filtering | IQL · Joint noise shift (9 tasks) | sum of normalized scores: 380.75 -> 654.85 | Helped |
| Filter source transitions using IGDF contrastive scores instead of training on mixed source and target data without filtering | IQL · Broken dynamics shift (9 tasks) | sum of normalized scores: 458.78 -> 538.31 | Helped |
| Filter source transitions using IGDF contrastive scores instead of training on mixed source and target data without filtering | IQL · Morphology shift (9 tasks) | sum of normalized scores: 431.65 -> 525.71 | Helped |
| Increase the amount of target-domain data (ablation) | IGDF and IQL · Cross-domain MuJoCo tasks | normalized score | Helped |
| Add importance weighting to filtered source data at a measured level (ablation) | IGDF+IQL · Cross-domain MuJoCo tasks | performance and smoothness | Helped |
| Use a too-small importance coefficient (ablation) | IGDF+IQL · Cross-domain MuJoCo tasks | performance | Hurt |
| Use an excessively large importance coefficient (ablation) | IGDF+IQL · Cross-domain MuJoCo tasks | training stability | Hurt |
Surprising findings
- Simply merging source and target data with IQL (w/o Aug) often performs worse than using only 10% of target data in the cross-domain setting.
- DARA performs poorly under larger dynamics gaps (morphology average -66.62%), attributed to unbounded dynamics-gap estimates.
Limitations
- Filters only by transition similarity and ignores trajectory quality, as acknowledged by the authors.
- Gains over baselines are not uniform; IGDF does not win every task (e.g., 17 of 18 rather than all on broken/morphology, and fewer wins on body mass/joint noise).
Experimental setup
Datasets: D4RL HalfCheetah D4RL Hopper D4RL Walker2d source-domain datasets collected in modified MuJoCo environments
Benchmarks: D4RL HalfCheetah body mass/joint noise/broken/morphology shifts D4RL Hopper shifts D4RL Walker2d shifts
Metrics: normalized average score average performance change relative to 100% target data
Baselines: DARA SRPO BOSA IQL CQL BCQ MOPO SPOT
Builds on: IQL InfoNCE DARA SRPO BOSA
Closest papers by meaning
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Cross-Domain Policy Adaptation by Capturing Representation Mismatch (ICML 2024 · similarity 0.90)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.88)
- Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.88)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.