Research map / Reinforcement learning / Offline RL
A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A2PO, which uses an advantage-conditioned CVAE to disentangle behavior policies in mixed-quality offline datasets and optimizes an advantage-aware TD3+BC-style policy toward high advantage values.
Problem
Offline RL from mixed-quality datasets collected by multiple behavior policies suffers from constraint conflict: uniform policy constraints treat conflicting actions with different returns equally, and advantage-weighted methods shrink data diversity.
Approach
Condition a CVAE on state plus a normalized advantage value (computed from Q and V critics) so it models each behavior policy's action distribution separately; the agent policy is conditioned on the maximum advantage (xi=1) to optimize returns while a BC term constrains it to in-sample actions matching each sample's advantage condition.
Key results
- A2PO achieves a Gym total of 1563.3 vs 1376.9 for the next-best CQL+AW, over 21% higher overall.
- On random-expert halfcheetah, A2PO scores 90.3 vs 87.3 for AWAC and 86.1 for Diffusion-QL; on random-medium hopper it scores 62.1 vs 22.6 for CQL+AW.
- Maze2d total of 609.5 vs 485.0 for Diffusion-QL; Kitchen total 209.2 vs 207.1 for Diffusion-QL and Adroit total 153.9 vs 130.1 for Diffusion-QL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A2PO instead of the strongest reported baseline | D4RL Gym tasks, total · Gym tasks (total) | normalized score: 1376.9 -> 1563.3 | Helped |
| Use A2PO instead of the strongest reported baseline | D4RL Maze tasks · Maze tasks (total) | test return: 485.0 -> 609.5 | Helped |
| Use A2PO instead of the strongest reported baseline | D4RL Kitchen tasks · Kitchen tasks (total) | test return: 207.1 -> 209.2 | Helped |
| Use A2PO instead of the strongest reported baseline | D4RL Adroit tasks · Adroit tasks (total) | test return: 130.1 -> 153.9 | Helped |
| Replace continuous advantage conditioning with a fixed ξ=1 for all samples (ablation) | A2PO · Gym random-expert halfcheetah | test return: 90.3 -> 5.8 | Hurt |
| Replace continuous advantage conditioning with discrete ξ, ε=0.1 (ablation) | A2PO · Gym random-expert halfcheetah | test return: 90.3 -> 21.9 | Hurt |
| Replace continuous advantage conditioning with discrete ξ, ε=0.0 (ablation) | A2PO · Gym medium-replay hopper | test return: 101.6 -> 96.9 | Hurt |
| Replace continuous advantage conditioning with discrete ξ, ε=0.1 (ablation) | A2PO · Gym random-expert walker2d | test return: 109.1 -> 109.6 | Mixed |
| Add BC regularization to the actor loss (ablation) | A2PO agent · Gym medium-replay halfcheetah | test return: 46.8 -> 44.8 | Helped |
| Add BC regularization to the actor loss (ablation) | A2PO agent · Gym medium halfcheetah | test return: 46.8 -> 47.1 | Helped |
| Use the optimal CVAE action as the agent output constraint instead of the original MSE alignment (ablation) | A2PO agent · Gym tasks | test return | Mixed |
| Increase the fixed advantage input during evaluation (ablation) | A2PO policy · Gym medium-replay halfcheetah | test return: 4.7 -> 21.5 -> 44.8 | Helped |
Surprising findings
- A fixed advantage input xi=1 (no advantage awareness) performs significantly worse, especially on random-expert mixed datasets.
- The CVAE policy alone is generally worse than the full agent policy, except on a few random-quality tasks such as hopper-random and walker2d-random.
Limitations
- CVAE training adds substantial time overhead compared with lightweight baselines like IQL.
- Best discrete advantage thresholds and alpha values are tuned per task using online evaluation, indicating sensitivity to hyperparameters.
- Results on Gym use the paper's own constructed mixed-quality datasets (random-medium, random-expert, random-medium-expert), which are not standard D4RL.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL maze2d D4RL antmaze D4RL kitchen D4RL adroit
Benchmarks: D4RL Gym D4RL Maze2d D4RL AntMaze D4RL Kitchen D4RL Adroit
Metrics: normalized score total score
Baselines: BC BCQ TD3+BC CQL MOPO EQL Diffusion-QL AWAC IQL CQL+AW LAPO %BC
Builds on: TD3+BC LAPO CVAE CQL Decision Transformer
Closest papers by meaning
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.94)
- Safe Offline Reinforcement Learning using Behavior Regularisation and Latent Feasibility-Guidance (NeurIPS 2026 · similarity 0.91)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.89)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.