Research map / Reinforcement learning / Offline RL

Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting

ICML 2024 · Reinforcement learning › Offline RL · approach: Behavior regularization for offline RL

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes adversarial data splitting (ADS), a min-max meta-learning framework that adversarially splits offline RL data into train/validation subsets to improve generalization near OOD boundaries, applied on top of CQL, TD3+BC and MCQ.

Problem

Offline RL suffers from OOD overestimation and overly conservative learning from suboptimal data; existing methods explicitly delimit OOD action boundaries, which impedes realistic value estimates for data near those boundaries.

Approach

Simulate distribution shift by splitting the offline dataset into train/validation subsets, then solve a min-max meta-learning problem: train on the train-subset to minimize validation loss while adversarially selecting the hardest (maximum-shift) validation split; stratified sampling via GMM clusters keeps subsets similar, and the loss is blended with CQL's conservative term using a support-based

Key results

• CQL+ADS improves over CQL on halfcheetah-medium from 49.4 to 73.9 and hopper-medium from 59.1 to 101.0 (normalized score).
• MCQ+ADS reaches 103.0 on hopper-medium and 105.0 on hopper-medium-replay, competitive with PRDC and STR on the average table (average 90.5 vs PRDC 90.1 across the listed datasets).
• On AntMaze umaze-diverse, CQL+ADS raises the score from 10.2 to 82.0 and TD3+BC+ADS from 47.0 to 90.0.

Measured effects reported in the paper

Surprising findings

• ADS-augmented CQL achieves a larger gain than replacing conservatism with pseudo Q-values, suggesting generalization is more beneficial than assigning pseudo Q-values.
• Increasing the validation-subset ratio hurts performance, as too little training data hampers finding good meta-parameters.

Limitations

• Applicability to discrete control tasks is not explored.
• Sparse-reward AntMaze gains are limited and the authors do not claim universal effectiveness on sparse-reward tasks.
• Convergence theorem relies on mild conditions and the generalization bound constant is treated as a constant.

Experimental setup

Datasets: D4RL halfcheetah-medium D4RL hopper-medium D4RL walker2d-medium D4RL medium-replay D4RL medium-expert D4RL antmaze

Benchmarks: D4RL MuJoCo gym v2 D4RL AntMaze

Metrics: normalized score absolute Q-value bias

Baselines: CQL TD3+BC MCQ BC IQL POR PRDC STR

Builds on: CQL TD3+BC MCQ MAML-style meta-learning (Li et al. 2018 / Qiao et al. 2020)

Closest papers by meaning

• Improving Generalization in Offline Reinforcement Learning via Latent Distribution Representation Learning (AAAI 2025 · similarity 0.92)
• Doubly Mild Generalization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
• Generative OOD-regularized Model-based Policy Optimization (NeurIPS 2026 · similarity 0.89)
• The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
• Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
• Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.88)