Research map / Reinforcement learning / Policy gradient and actor-critic

Deep Double Q-learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts Double Q-learning to deep RL by training two Q-functions with reciprocal bootstrapping and dataset partitioning, yielding DDQL, which outperforms Double DQN on Atari-57 while reducing overestimation.

Problem

Double DQN only partially decouples action selection and evaluation (single trained Q-network plus target network), so overestimation persists in deep value-based RL.

Approach

Explicitly train two Q-functions (shared-trunk two-head DDQL, or separate-network DN-DDQL) that bootstrap off each other via target networks, sampling distinct minibatches per Q-function, combined with lower replay ratio (1/8), longer target update intervals, identical initialization, and simultaneous updates for stability.

Key results

• DDQL improves IQM of human-normalized score by 31% over Double DQN across 57 Atari games and outperforms Double DQN on 47 of 57 games.
• DDQL reduces overestimation relative to Double DQN in 42 of 57 environments.
• DN-DDQL, with separate networks, underestimates and performs 23% better than Double DQN in IQM, slightly worse than DDQL.

Surprising findings

• Separate-network DN-DDQL drives overestimation to underestimation, yet performs slightly worse than the shared-trunk DDQL.
• Doubling the replay ratio harms DDQL, and a lower replay ratio does not benefit Double DQN, indicating DDQL's gains are not just from fewer updates.

Limitations

• Overestimation reduction does not uniformly translate into performance gains; DN-DDQL underestimates and performs worse.
• Evaluated only on Atari 2600 games with a DQN-style setup and PFRL implementation.

Experimental setup

Datasets: Atari 2600 (Arcade Learning Environment) Atari-57 Ablation-11 subset

Benchmarks: Atari-57 Ablation-11

Metrics: interquartile mean of human-normalized score per-game AUC improvement in HNS overestimation (Q-hat minus discounted return)

Baselines: Double DQN DQN DN-DDQL

Builds on: Double Q-learning Double DQN DQN PFRL

Closest papers by meaning

• Deep Reinforcement Learning with Double Q-Learning (AAAI 2016 · similarity 0.93)
• ADDQ: Adaptive distributional double Q-learning (ICML 2025 · similarity 0.89)
• Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.87)
• Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.86)
• Confounding Robust Deep Reinforcement Learning: A Causal Approach (NeurIPS 2025 · similarity 0.86)
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.85)