Research map / Reinforcement learning / Offline RL
Improving Generalization in Offline Reinforcement Learning via Latent Distribution Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAD, a min-max adversarial framework that clusters offline RL data into latent distributions, adversarially finds worst-case gaps, and learns invariant representations to improve OOD generalization on D4RL.
Problem
Distribution shift in offline RL causes poor generalization to out-of-distribution data; prior domain-adaptation methods simulate shifts with heuristic dataset splits, giving limited diversity of distribution gaps.
Approach
Treat offline data as containing multiple latent distributions (via GMM clustering of state-action pairs), adversarially maximize gaps between them with a gradient reverse layer, then learn invariant representations by bridging these gaps; a generalization error bound for multi-distribution sources supports it.
Key results
- TD3BC+LAD and MCQ+LAD improve over base TD3BC and MCQ and generally exceed ADS variants on D4RL MuJoCo tasks (e.g., Hopper-ME 112.7 vs 105.8 for MCQ+ADS... MCQ+LAD Hopper-MR 105.8).
- MCQ+LAD achieves an average normalized score of 95.6 across nine D4RL gym datasets, versus 92.3 for CQL+ADS and 90.1 for PRDC.
- Ablations show removing the distribution learner or the fine-tuning step degrades performance; K=5 chosen as default.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LAD to TD3BC | TD3BC · D4RL HalfCheetah-M, Hopper-M, Walker2d-M, HalfCheetah-MR, Hopper-MR, Walker2d-MR, HalfCheetah-ME, Hopper-ME, Walker2d-ME | normalized score | Mixed |
| Add LAD to MCQ | MCQ · D4RL HalfCheetah-M, Hopper-M, Walker2d-M, HalfCheetah-MR, Hopper-MR, Walker2d-MR, HalfCheetah-ME, Hopper-ME, Walker2d-ME | normalized score | Helped |
| Add LAD to MCQ | MCQ · D4RL HalfCheetah-ME | normalized score: 80.1 ± 3.8 -> 101.5 ± 1.2 | Helped |
| Remove the latent distribution learner L(s, a) (ablation) | LAD · HalfCheetah-MR | normalized score | Hurt |
| Replace the fine-tuning target y′ with the original target y (ablation) | LAD · HalfCheetah-MR | normalized score | Hurt |
Limitations
- Iterative min-max procedure limits scalability to large datasets in the pre-processing (clustering) phase
- Evaluated only on D4RL MuJoCo locomotion tasks
Experimental setup
Datasets: D4RL halfcheetah-medium D4RL hopper-medium D4RL walker2d-medium D4RL halfcheetah-medium-replay D4RL hopper-medium-replay D4RL walker2d-medium-replay D4RL halfcheetah-medium-expert D4RL hopper-medium-expert D4RL walker2d-medium-expert
Benchmarks: D4RL
Metrics: normalized score
Baselines: BC IQL CQL SPOT POR PRDC STR Diffusion-QL CQL+ADS TD3BC TD3BC+ADS MCQ
Builds on: ADS TD3BC MCQ Ganin et al. gradient reverse layer domain-adversarial training
Closest papers by meaning
- Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting (ICML 2024 · similarity 0.92)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Doubly Mild Generalization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Generative OOD-regularized Model-based Policy Optimization (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.