Research map / Reinforcement learning / Policy gradient and actor-critic

ADDQ: Adaptive distributional double Q-learning

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Stabilizing deep actor-critic

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADDQ, which locally weights between Q-learning and double Q-learning updates using sample variances of return distributions from distributional RL, to control overestimation bias.

Problem

Overestimation bias in Q-learning slows convergence, while double Q-learning can underestimate; no rule exists for which works best in a given environment, so a locally adaptive control of the bias is needed.

Approach

Use the sample variance of the return distribution (from distributional RL) to compute state-local weights beta that interpolate between Q-learning and double Q-learning targets, favoring Q-learning where variance is low relative to other actions and double Q-learning where it is high.

Key results

• ADDQ shows much smaller Q-value estimation bias than Q-learning and double Q-learning in the grid world example.
• ADDQ is more stable than C51 and double C51 on Atari (never fails completely) and improves QR-DQN and quantile SAC variants, but does not reach clipped double estimators on MuJoCo.
• Theory: a lower bound on Q-learning overestimation under 1/visits step sizes, and the sample variance of the distributional estimate is scaled chi-squared with N-1 degrees of freedom.

Measured effects reported in the paper

Surprising findings

• On MuJoCo, ADDQ cannot reach the performance of clipped double critic estimation, so the method does not compete with TD3/SAC-style clipping.
• Double Q-learning can fail completely on some Atari environments, where the adaptive method avoids such failures.

Limitations

• Beta thresholds are hand-chosen hyperparameters and not adaptive.
• Not competitive with clipped double Q-learning or REDQ/TQC on MuJoCo.
• Theoretical analysis is limited to a simple tabular bandit with Gaussian rewards.

Experimental setup

Datasets: Atari (Arcade Learning Environment) MuJoCo tabular grid world two-sided bandit MDP

Benchmarks: Atari MuJoCo grid world two-sided bandit MDP

Metrics: learning curves RLiable probability of improvement normalized scores estimation bias

Baselines: Q-learning double Q-learning C51 QR-DQN clipped double Q-learning Maxmin Q-learning REDQ weighted double Q-learning SAC

Builds on: double Q-learning C51 QR-DQN TD3 clipped double Q-learning weighted double Q-learning

Closest papers by meaning

• Deep Double Q-learning (NeurIPS 2026 · similarity 0.89)
• Regularized Q-Learning (NeurIPS 2024 · similarity 0.87)
• Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.87)
• Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
• $Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training (NeurIPS 2025 · similarity 0.86)
• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.86)