Research map / Reinforcement learning / Offline RL
Exclusively Penalized Q-learning for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Exclusively Penalized Q-learning (EPQ), which applies CQL-style value penalties only to states/actions where policy actions are poorly covered by the data, with an optional prioritized dataset, to reduce underestimation bias in offline RL.
Problem
Penalty-based offline RL such as CQL over-penalizes well-covered actions, introducing unnecessary underestimation bias in the value function, which degrades performance as the penalty weight grows.
Approach
Scales the CQL penalty by a threshold-based penalty adaptation factor (average of a clipped exp(-(log beta - tau)) over policy actions) so that penalty is removed for actions well represented in the dataset, and adds a prioritized dataset beta^Q proportional to beta*exp(Q) to further shrink the penalty.
Key results
- EPQ achieves 1536.7 total on Mujoco tasks versus 1474.9 for the next best baseline (MCQ) and 1325.8 for CQL (reproduced).
- EPQ achieves 221.8 total on Adroit tasks and 426.1 on AntMaze tasks, versus 162.7 and 378.0 for the best baselines (MISA and IQL).
- EPQ reaches a normalized score of 100 on hopper-medium with about 30,633 s score runtime, while all other baselines fail to reach 100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EPQ instead of the strongest reported baseline, MCQ | Offline RL · Mujoco tasks total | normalized average return: 1474.9 -> 1536.7 | Helped |
| Use EPQ instead of the strongest reported baseline, MISA | Offline RL · Adroit tasks total | normalized average return: 162.7 -> 221.8 | Helped |
| Use EPQ instead of the strongest reported baseline, IQL | Offline RL · AntMaze tasks total | normalized average return: 378.0 -> 426.1 | Helped |
| Use the exclusive penalty instead of the CQL penalty | EPQ · Hopper-random, Hopper-medium, and HalfCheetah-medium | estimation bias | Helped |
| Add prioritized dataset (PD) (ablation) | EPQ · HalfCheetah-medium | normalized average return | Helped |
| Add the exclusive penalty without prioritized dataset (ablation) | CQL · Hopper-random, Hopper-medium, and HalfCheetah-medium | normalized average return | Helped |
| Vary the penalty-control threshold τ (ablation) | EPQ · Hopper-random, Hopper-medium, and HalfCheetah-medium | normalized average return | No clear effect |
| Use a higher penalty-control threshold (ablation) | EPQ · Hopper-medium | normalized average return | Helped |
| Use a lower penalty-control threshold (ablation) | EPQ · Hopper-random | normalized average return | Helped |
Surprising findings
- Reducing the CQL penalty alpha to nearly 0 diverges Q-functions on Hopper tasks, so simply lowering the penalty is not a fix.
- Performance of EPQ is reported as surprisingly insensitive to the threshold tau.
Limitations
- Results reported with best hyperparameter per task selected from ablations, so tuned comparison.
- Adroit CQL baseline reproduction underperforms the paper result, requiring a revised CQL.
- Runtime per gradient step is higher than CQL and IQL/MISA.
Experimental setup
Datasets: D4RL Mujoco datasets D4RL Adroit datasets D4RL AntMaze datasets
Benchmarks: D4RL Mujoco locomotion D4RL Adroit manipulation D4RL AntMaze
Metrics: normalized average return
Baselines: BC 10% BC TD3+BC Onestep RL IQL MCQ MISA CQL
Builds on: Conservative Q-learning (CQL) TD3+BC Implicit Q-learning Batch-constrained Q-learning
Closest papers by meaning
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.93)
- Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.91)
- Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning (ICLR 2025 · similarity 0.91)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.90)
- Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.