Research map / Reinforcement learning / Offline RL
Doubly Mild Generalization for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Doubly Mild Generalization (DMG), blending a mildly generalized max (actor-critic with KL-regularized policy toward advantage-reshaped behavior) with an in-sample expectile max in the Bellman target, with theory and D4RL results.
Problem
Offline RL suffers from extrapolation error and value overestimation due to over-generalization of value functions/policies to OOD actions, while in-sample methods avoid generalization entirely and may be overly conservative.
Approach
Allow mild action generalization (maximizing Q over a policy constrained near the dataset) and mild generalization propagation (discount on the generalized max reduced to lambda*gamma, blended with the in-sample expectile max), giving bounded overestimation and a performance lower bound.
Key results
- DMG achieves 1182.8 total on Gym locomotion vs 1033.1 for IQL and 1041.2 for OneStep RL (Dataset-v2, 5 seeds).
- DMG achieves 439.4 total on AntMaze vs 392.4 for IQL, with largest gains on large mazes (e.g., antmaze-l-d 58.8 vs 46.0).
- Online fine-tuning on AntMaze: DMG reaches 556.4 total after 1M online steps vs 490.4 for IQL; TD3 from scratch scores 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DMG instead of the strongest listed baseline | Gym locomotion offline RL · Gym locomotion total | normalized score: 1041.2 -> 1182.8 | Helped |
| Use DMG instead of IQL | AntMaze offline RL · AntMaze total | normalized score: 392.4 -> 439.4 | Helped |
| Add DMG to XQL | XQL · Gym locomotion total | normalized score: 725.3 -> 792.7 | Helped |
| Add DMG to SQL | SQL · Gym locomotion total | normalized score: 747.5 -> 804.2 | Helped |
| Increase mixture coefficient λ, allowing stronger generalization propagation (ablation) | DMG · Offline RL tasks | performance | Mixed |
| Decrease penalty coefficient ν, allowing broader action generalization (ablation) | DMG · Offline RL tasks | performance | Mixed |
| Fine-tune offline DMG for 1M online steps instead of stopping after offline training | DMG · AntMaze total | normalized score: 439.4 -> 556.4 | Helped |
Surprising findings
- Mild generalization beyond the dataset improves over in-sample optimal policies, contrary to the prevailing non-generalization view.
- Training from scratch with TD3 fails entirely on AntMaze, whereas offline-pretrained DMG succeeds in online fine-tuning.
Limitations
- When the function approximator generalizes well over the entire action space, DMG may underperform full-generalization methods due to conservatism.
- Theoretical guarantees rely on Lipschitz and oracle generalization assumptions that may not hold in practice.
Experimental setup
Datasets: D4RL Gym-MuJoCo locomotion datasets D4RL AntMaze datasets
Benchmarks: D4RL Gym-MuJoCo locomotion D4RL AntMaze
Metrics: normalized score total score Q value estimates
Baselines: BC BCQ BEAR DT AWAC OneStep RL TD3BC CQL IQL XQL SQL TD3
Builds on: IQL TD3BC OneStep RL in-sample Q learning Polyak averaging target networks
Closest papers by meaning
- Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting (ICML 2024 · similarity 0.92)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Provable Zero-Shot Generalization in Offline Reinforcement Learning (ICML 2025 · similarity 0.91)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.90)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.