Research map / Reinforcement learning / Offline RL
Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QDQ, which estimates Q-value uncertainty from a consistency-model-learned conditional Q-distribution and penalizes OOD-action Q-targets, with an uncertainty-aware objective mixing optimistic and pessimistic Bellman residuals for offline RL.
Problem
Q-value overestimation for out-of-distribution actions in offline RL, where pessimistic penalties are either applied uniformly (over-conservatism) or based on inaccurate uncertainty estimates.
Approach
Learn the behavior policy's conditional Q-value distribution with a one-step consistency model trained on sliding-window truncated returns; use the variance of bootstrap samples as an uncertainty measure to penalize Q-targets for high-uncertainty (likely OOD) actions, and mix standard and pessimistic Bellman residuals in the Q loss to reduce conservatism.
Key results
- QDQ reaches a total of 848.0 on the nine Gym-MuJoCo medium/medium-replay/medium-expert tasks, versus 841.1 for EDAC and 797.4 for MCQ.
- On AntMaze, QDQ reaches a total of 400.1 versus 378.0 for IQL, with 98.6 on umaze and 81.5 on medium-play.
- QDQ is weaker than the best baseline on several tasks, e.g., large-play (35.6 vs 39.6 for IQL) and large-diverse (31.2 vs 47.5 for IQL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QDQ instead of EDAC | halfcheetah-medium-v2 · halfcheetah-medium-v2 | normalized score: 65.9 -> 74.1 ±1.7 | Helped |
| Use QDQ instead of EDAC | hopper-medium-v2 · hopper-medium-v2 | normalized score: 101.6 -> 99.0 ±0.3 | Hurt |
| Use QDQ instead of EDAC | walker2d-medium-v2 · walker2d-medium-v2 | normalized score: 92.5 -> 86.9{\pm0.08} | Hurt |
| Use QDQ instead of EDAC | halfcheetah-medium-replay-v2 · halfcheetah-medium-replay-v2 | normalized score: 61.3 -> 63.7 ±2.9 | Helped |
| Use QDQ instead of MCQ | hopper-medium-replay-v2 · hopper-medium-replay-v2 | normalized score: 101.6 -> 102.4 ±0.28 | Helped |
| Use QDQ instead of MCQ | walker2d-medium-replay-v2 · walker2d-medium-replay-v2 | normalized score: 91.3 -> 93.2 ±1.1 | Helped |
| Use QDQ instead of EDAC | halfcheetah-medium-expert-v2 · halfcheetah-medium-expert-v2 | normalized score: 106.3 -> 99.3{\pm1.7} | Hurt |
| Use QDQ instead of IQL | AntMaze medium-play-v0 · AntMaze medium-play-v0 | normalized score: 71.2 -> 81.5{\pm3.6} | Helped |
| Use QDQ instead of IQL | AntMaze medium-diverse-v0 · AntMaze medium-diverse-v0 | normalized score: 70.0 -> 85.4{\pm4.2} | Helped |
| Use QDQ instead of IQL | AntMaze large-play-v0 · AntMaze large-play-v0 | normalized score: 39.6 -> 35.6{\pm5.4} | Hurt |
| Use QDQ instead of IQL | AntMaze large-diverse-v0 · AntMaze large-diverse-v0 | normalized score: 47.5 -> 31.2{\pm4.5} | Hurt |
Surprising findings
- The paper reports that QDQ underperforms IQL on the AntMaze large-play and large-diverse tasks.
Limitations
- Some AntMaze large tasks are weaker than IQL.
- Hyperparameters alpha, beta, gamma are tuned per task and sensitivity to them is only partially analyzed.
Experimental setup
Datasets: D4RL Gym-MuJoCo (v2) D4RL AntMaze (v0)
Benchmarks: D4RL Gym-MuJoCo D4RL AntMaze
Metrics: normalized score
Baselines: BC AWAC DT TD3+BC CQL IQL UWAC EDAC
Builds on: consistency models TD3+BC UWAC CQL
Closest papers by meaning
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.92)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.89)
- C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning (ICML 2025 · similarity 0.88)
- Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.