Research map / Reinforcement learning / Policy gradient and actor-critic
Simplifying Deep Temporal Difference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that LayerNorm plus l2 regularisation yields provably convergent TD without target networks or replay buffers, and uses this to build PQN, a parallelised online deep Q-learning method evaluated on Atari, Craftax, Smax, Overcooked and Hanabi.
Problem
Deep TD and Q-learning are unstable under off-policy data and nonlinear function approximation, and standard fixes (target networks, large replay buffers) hurt sample efficiency and memory, especially in pure-GPU vectorised RL where off-policy Q-learning lacks a baseline.
Approach
A Jacobian-based TD stability criterion shows LayerNorm bounds the off-policy and nonlinear terms of the TD Jacobian; adding l2 regularisation on the final layer makes the criterion hold for large width, enabling target-network-free, buffer-free Q-learning. PQN applies this with vectorised environments that supply online, parallel samples, using lambda-returns and minibatch epochs.
Key results
- PQN reaches the median score of Rainbow on Atari-57 with an additional 100M frames (about 30 minutes of training), claimed to be about 50x faster than DQN-style pipelines.
- On Craftax with an RNN, PQN achieves 16% versus 15.3% for PPO-RNN, with similar training time.
- PQN with VDN outperforms MAPPO and QMix on Smax and learns coordination in about 10 minutes versus QMix's 1 hour.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PQN instead of the traditional DQN pipeline | Atari · Atari | training speed: 50x | Helped |
| use PQN instead of the replay-buffer baseline | Smax · Smax | training speed: 6x | Helped |
| use PQN instead of the replay-buffer baseline | Craftax · Craftax | training speed: 6x | Helped |
| use PQN instead of the R2D2 multi-threaded implementation | Hanabi · Hanabi | training speed: 4x | Helped |
| use PQN RNN instead of PPO-RNN | Craftax · Craftax | score: 15.3% -> 16% | Helped |
| add LayerNorm (ablation) | PQN · Atari-10 | performance | Helped |
| apply BatchNorm through the network (ablation) | PQN · Atari-10 | performance | Hurt |
| add the additional tricks from CrossQ (ablation) | PQN · Atari-10 | performance | Hurt |
| batch-normalise the input before the first hidden layer (ablation) | PQN RNN · Craftax | performance | Helped |
| use BatchNorm only on the input to the first layer (ablation) | PQN RNN · Craftax | performance | Helped |
| use λ=0.65 instead of λ=0 (ablation) | PQN · Atari-10 | performance | Helped |
| add a standard-sized replay buffer of 1M experiences (ablation) | PQN · Craftax | training time: ∼6x longer | Hurt |
Surprising findings
- LayerNorm without l2 regularisation cannot fully stabilise TD in all domains; Baird's counterexample required l2 regularisation on final-layer weights.
- Applying CrossQ-style tricks to PQN further worsened its performance.
Limitations
- Theory covers wide LayerNorm networks and a residual off-policy term; l2 regularisation must be used sparingly since large values pull fixed points toward zero.
- Epsilon-greedy exploration is too simple for hard-exploration Atari games.
- Speedup claims are compared against implementations with different hardware and pipelines, and DQN training time was optimistically estimated.
Experimental setup
Datasets: Atari (ALE) 57 games Craftax Smax (SMACv2) Overcooked Hanabi Baird's counterexample MinAtar Classic Control
Benchmarks: Baird's counterexample Atari-10 Atari-57 Craftax Smax Overcooked Hanabi
Metrics: median human-normalised score IQM score win rate wall-clock time memory saved
Baselines: PPO Rainbow DQN Double DQN Prioritised DDQN Ape-X R2D2 PPO-RNN QMix MAPPO VDN CrossQ
Builds on: DQN Q-learning Retrace VDN CrossQ
Closest papers by meaning
- A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning (NeurIPS 2025 · similarity 0.88)
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.88)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.87)
- MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.