Research map / Reinforcement learning / Policy gradient and actor-critic
Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaQN, an ensemble of Q-networks trained with different hyperparameters that share as target the member with the smallest approximation error, enabling on-the-fly hyperparameter selection without extra environment samples.
Problem
Deep RL is highly sensitive to hyperparameters, and existing AutoRL methods consume extra environment samples for selection and borrow AutoML ideas that ignore RL non-stationarity.
Approach
Train K online Q-networks with different hyperparameters and, at each target update, select the one with the lowest empirical approximation (Bellman projection) loss as the shared target for all networks; the selection uses training loss rather than environment rollouts, and the approach is orthogonal to the critic-based algorithm.
Key results
- On MuJoCo with 16 hyperparameter sets, AdaSAC is an order of magnitude more sample-efficient than grid search and yields the highest AUC, outperforming 13 of 16 individual SAC runs in final performance.
- On Atari with an infinite hyperparameter space (18 activations, 24 optimizers, 4 losses), AdaDQN outperforms SEARL, random search and DEHB at 40M frames and keeps improving to 100M frames.
- AdaDQN with the target chosen by minimum loss outperforms the best individual Lunar Lander architecture in AUC, whereas the max-selection variant performs as badly as the worst agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select the target network online from diverse architectures using minimum approximation error | AdaDQN · Lunar Lander | AUC | Helped |
| Set ε_b = 0, eliminating random action-network selection (ablation) | AdaDQN · Lunar Lander | AUC | Hurt |
| Select the target network randomly instead of by minimum approximation loss (ablation) | AdaDQN · Lunar Lander | performance | Hurt |
| Replace minimum-loss target selection with maximum-loss selection (ablation) | AdaDQN · Lunar Lander | performance | Hurt |
| Select hyperparameters online from 16 configurations during training | AdaSAC · MuJoCo environments | AUC | Helped |
| Select hyperparameters online from 16 configurations during training | AdaSAC · MuJoCo environments | sample efficiency | Helped |
| Select the target network randomly instead of by approximation loss (ablation) | AdaSAC · HalfCheetah | performance | Hurt |
| Vary critic architecture, learning rate, or optimizer as the hyperparameter search dimension (ablation) | AdaSAC · MuJoCo environments | performance | Helped |
| Select among three activation functions online (ablation) | AdaSAC · MuJoCo environments | performance | Helped |
| Select online among activation functions, architectures, or Adam epsilon values (ablation) | AdaDQN · Atari 2600, finite hyperparameter space | performance | No clear effect |
| Select targets online using approximation loss in a broad hyperparameter search | AdaDQN · Atari 2600, infinite hyperparameter space | IQM return | Helped |
| Select the discount factor online | AdaDQN · Atari 2600, discount factor selection | performance | No clear effect |
Surprising findings
- Selecting the target by maximum approximation error (AdaDQN-max) performs as badly as the worst individual agent.
- Random target selection (RandSAC/RandDQN) can underperform when one ensemble member performs poorly, yet the adaptive min-loss selection avoids this.
Limitations
- Cannot handle environment properties such as the reward function, only agent hyperparameters.
- Training time and memory grow with the number of Q-functions in the ensemble.
- Convergence proof is only for a tabular setting; theory relies on an unbiased empirical Bellman operator assumption with infinite data.
Experimental setup
Datasets: Lunar Lander MuJoCo Atari 2600
Benchmarks: Lunar Lander MuJoCo Atari 2600
Metrics: interquartile mean return area under the curve worst-seed return
Baselines: DQN grid search random search RandSAC RandDQN SEARL DEHB Meta-gradient DQN SAC
Builds on: DQN Soft Actor-Critic SEARL Meta-gradient reinforcement learning REDQ
Closest papers by meaning
- Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.88)
- Stochastic Q-learning for Large Discrete Action Spaces (ICML 2024 · similarity 0.88)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.88)
- Deep Double Q-learning (NeurIPS 2026 · similarity 0.87)
- Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.