Research map / Reinforcement learning / Offline RL
Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A2PR, which trains a VAE weighted by advantage to generate high-advantage actions that augment the behavior policy used for TD3+BC-style policy regularization in offline RL, reporting state-of-the-art D4RL results.
Problem
Policy regularization in offline RL is unnecessarily conservative because it forces the learned policy to imitate all dataset actions, including suboptimal ones, which hampers improvement especially on low-quality datasets.
Approach
Train a VAE with an advantage-thresholded reconstruction term (elevating positive behavior learning) so it generates high-advantage actions; then regularize the policy toward whichever of the dataset action or VAE action has higher advantage, adaptively falling back to the policy itself when no action is advantageous.
Key results
- A2PR reports a Gym average of 944.27 and total average 1422.47 across D4RL Gym and AntMaze tasks, versus 1106.5 for IQL and 1324.7 for PRDC (best reported baseline total average).
- A2PR achieves the best mean score in 16 of 18 Gym tasks per the paper's claim, e.g., halfcheetah-medium-expert 98.25 vs 94.5 for PRDC.
- On the 99% random / 1% expert mixed dataset, A2PR outperforms TD3+BC and IQL, with the paper stating a substantial gap, particularly on halfcheetah.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A2PR instead of competing baselines | offline RL policies · D4RL Gym and AntMaze tasks | normalized score | Mixed |
| Remove elevated positive behavior learning (EPBL) (ablation) | A2PR on halfcheetah-medium-v2 · halfcheetah-medium-v2 | normalized score | Hurt |
| Replace the improved VAE with a standard VAE (ablation) | A2PR on halfcheetah-medium-v2 · halfcheetah-medium-v2 | normalized score | Hurt |
| Remove adaptive advantage policy constraint (AAPC) (ablation) | A2PR on halfcheetah-medium-v2 · halfcheetah-medium-v2 | normalized score | Hurt |
| Set advantage threshold to 0 instead of varying it (ablation) | A2PR · D4RL Hopper, HalfCheetah, and Walker2d datasets | normalized score | Mixed |
| Use A2PR rather than TD3+BC | offline policy on mixed random/expert data · 99% random policy data and 1% expert policy data | normalized score | Helped |
| Use A2PR rather than TD3+BC | offline policy on low-quality maze data · Multiple Target Maze | target return | Helped |
| Add Gaussian state-input noise during evaluation | A2PR · Hopper, HalfCheetah, and Walker2d datasets | normalized score | Hurt |
| Use A2PR with SAC rather than its TD3 variant | A2PR · HalfCheetah datasets | normalized score | Hurt |
Surprising findings
- Table 2 reports A2PR's total average (1422.47) but the Gym average row values and AntMaze averages are reported inconsistently with per-task numbers in some cases, so the aggregate claims should be checked against per-task results.
Limitations
- Sampling a single VAE action can make learning unstable because quality of a single sample is hard to ensure.
- The VAE's limited expressiveness as a behavior policy model may restrict identifying high-advantage actions; the authors suggest diffusion models as future work.
- Theoretical guarantees rely on assumptions such as Lipschitz continuity and approximately equal state visitation distributions, with some steps only approximate.
Experimental setup
Datasets: D4RL halfcheetah v2 D4RL hopper v2 D4RL walker2d v2 D4RL antmaze v2 multi-target maze (custom) 99% random 1% expert mixed dataset
Benchmarks: D4RL Gym D4RL AntMaze multi-target maze mixed random-expert dataset
Metrics: normalized D4RL score true vs estimated Q-value training time mean advantage
Baselines: TD3+BC BCQ BEAR CQL IQL AW OAP PRDC
Builds on: TD3+BC IQL PRDC TD3
Closest papers by meaning
- A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective (NeurIPS 2024 · similarity 0.94)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.94)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.91)
- An Implicit Trust Region Approach to Behavior Regularized Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.91)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.