Research map / Reinforcement learning / Offline RL
Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DORA, an offline meta-RL method that trains a recurrent context encoder with an information-bottleneck objective (contrastive distortion loss plus a KL-based debias loss against behavior-policy actions) and then runs CQL on the encoder's representations to adapt online to non-stationary MuJoCo dynamics.
Problem
Offline meta-RL policies must adapt online to non-stationary dynamics, but a context encoder trained on limited offline data confuses dynamics changes with behavior-policy shifts, causing context misassociations and failure when the learned policy collects context.
Approach
Apply the information bottleneck principle to offline context encoder learning: a contrastive lower bound maximizes mutual information between the encoding and the task, while a KL-divergence upper bound minimizes mutual information between the encoding and behavior-policy actions, so the encoder tracks dynamics rather than the behavior policy and can infer dynamics from the most recent short hist
Key results
- In non-stationary dynamics, DORA reaches 71.61 on Cheetah-gravity versus 65.42 for offline ESCP and 54.08 for CORRO.
- DORA outperforms baselines on 5 of 6 MuJoCo tasks in IID and OOD stationary dynamics; offline ESCP is better on Hopper-gravity (78.70 vs 74.68 IID).
- Removing the distortion loss prevents DORA from converging; the debias loss improves performance in ablations on 3 environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DORA instead of Offline ESCP | offline meta-RL policy · Cheetah-gravity non-stationary | normalized return: 65.42 $\pm$ \,\,\,6.54 -> 71.61 $\pm$ \,\,\,7.56 | Helped |
| use DORA instead of Offline ESCP | offline meta-RL policy · Pendulum-gravity non-stationary | normalized return: 85.92 $\pm$ 16.37 -> 97.11 $\pm$ 16.91 | Helped |
| use DORA instead of CORRO | offline meta-RL policy · Walker-gravity non-stationary | normalized return: 33.62 $\pm$ 14.86 -> 40.46 $\pm$ 19.04 | Helped |
| use DORA instead of Offline ESCP | offline meta-RL policy · Hopper-gravity non-stationary | normalized return: 54.29 $\pm$ 25.39 -> 49.74 $\pm$ 23.74 | Hurt |
| use DORA instead of Offline ESCP | offline meta-RL policy · Cheetah-dof non-stationary | normalized return: 80.72 $\pm$ \,\,\,8.49 -> 91.83 $\pm$ 10.57 | Helped |
| use DORA instead of Offline ESCP | offline meta-RL policy · Cheetah-torso non-stationary | normalized return: 53.99 $\pm$ 16.09 -> 60.13 $\pm$ 13.65 | Helped |
| add the debias loss (ablation) | DORA · three randomly chosen environments | normalized return | Helped |
| remove the distortion loss (ablation) | DORA · Cheetah-gravity | normalized return | Hurt |
| use RNN history length 4 or 16 instead of 8 (ablation) | DORA · three MuJoCo environments | normalized return | No clear effect |
| reduce the number of training tasks from 10 to 8 (ablation) | DORA · Cheetah-gravity non-stationary | normalized return: 71.61 $\pm$ 7.56 -> 65.48 $\pm$ 10.58 | Hurt |
| reduce the number of training tasks from 10 to 5 (ablation) | DORA · Cheetah-gravity non-stationary | normalized return: 71.61 $\pm$ 7.56 -> 54.92 $\pm$ 17.71 | Hurt |
| halve the offline dataset size (ablation) | DORA · Cheetah-gravity non-stationary | normalized return: 71.61 $\pm$ 7.56 -> 66.28 $\pm$ 7.85 | Hurt |
Surprising findings
- DORA cannot converge without the distortion loss, so the debias loss alone is insufficient.
- Prompt-DT performs poorly, which the authors attribute to the transformer lacking a well-defined prompt in the offline non-stationary setting.
Limitations
- The debias loss weight beta must be tuned per task (0.2 to 1.0 in the reported configs), and results are sensitive to beta.
- Evaluated only on MuJoCo with gravity, dof-damping and torso-mass perturbations, with 10 training tasks per environment.
- Results with fewer training tasks drop substantially (e.g., 54.92 with 5 tasks).
Experimental setup
Datasets: MuJoCo offline datasets collected from SAC replay buffers per training task
Benchmarks: HalfCheetah-v3 gravity/dof/torso (MuJoCo) Walker2d-v3 gravity Hopper-v3 gravity InvertedDoublePendulum-v2 gravity
Metrics: normalized return standard deviation over 5 seeds
Baselines: Offline ESCP FOCAL CORRO Prompt-DT
Builds on: CQL InfoNCE PEARL ESCP FOCAL
Closest papers by meaning
- Behavior Prior Representation learning for Offline Reinforcement Learning (ICLR 2023 · similarity 0.89)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.87)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.