Research map / Reinforcement learning / Policy gradient and actor-critic
Learning the Target Network in Function Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lookahead-Replicate (LR), which replaces parameter-copying of the target network with gradient-based function-space matching of target and online value functions, with convergence analysis and improved Rainbow results on Atari.
Problem
Target-network updates in value-function approximation enforce parameter equivalence between online and target networks, which is stricter than needed for learning a single value function.
Approach
Replace the parameter-copy target update with a Replicate step that minimizes the value (or distributional cross-entropy) discrepancy between target and online networks via gradient descent, so the two parameter sets may differ while their value functions match; a Lookahead step performs the Bellman bootstrapping update.
Key results
- LR with all-action variant outperforms Rainbow with frequency-based and Polyak target updates on 6 Atari games, averaged over 5 seeds.
- Performance vs K_R shows an inverted-U shape; intermediate K_R=800 (0.1 x K_L) performs best among tested values.
- On all 55 Atari games, LR outperforms Rainbow on both frame-based and gradient-step-based comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the LR target-network update instead of Rainbow's standard target-network update | Rainbow · Atari (55 games) | median human-normalized score | Helped |
| Use the all-action variant instead of the one-action variant (ablation) | LR · Atari (6 games) | median human-normalized score | Helped |
| Vary the number of target-network updates in the Replicate step (ablation) | LR · Atari (6 games) | area under the curve | Mixed |
| Tune the hard target-update frequency (ablation) | Rainbow · Atari | performance | No clear effect |
| Use Polyak-based instead of frequency-based target updates (ablation) | Rainbow · Atari (6 games) | performance | No clear effect |
Surprising findings
- Polyak target update is roughly as effective as the frequency-based update in Rainbow.
- Learned online and target parameters differ in norm under LR, indicating solutions with theta differing from w.
Limitations
- Convergence theory relies on strong convexity and Lipschitz assumptions, shown in linear settings.
- Only a 6-game subset was used for hyperparameter ablation; K_R tuned on it.
- Learning rate for Replicate not tuned.
Experimental setup
Datasets: Atari 2600 games
Benchmarks: Atari 2600 (6-game subset) Atari 55 games
Metrics: human-normalized median score area under curve parameter norm
Baselines: Rainbow (frequency-based target update) Rainbow (Polyak target update)
Builds on: Rainbow DQN C51 Dopamine
Closest papers by meaning
- Target-Aligned Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning (NeurIPS 2025 · similarity 0.87)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.87)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
- Stochastic Q-learning for Large Discrete Action Spaces (ICML 2024 · similarity 0.86)
- In value-based deep reinforcement learning, a pruned network is a good network (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.