Research map / Reinforcement learning / Policy gradient and actor-critic
The Phenomenon of Policy Churn
Tom Schaul, Andre Barreto, John Quan, Georg Ostrovski
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| act with epsilon=0 instead of decaying epsilon-greedy (ablation) | DoubleDQN · Atari 15 games | return | No clear effect |
| act with the target network (updated every 120 000 frames) to remove churn from behaviour (ablation) | DoubleDQN with epsilon-greedy · Atari 15 games | return | Hurt |
| remove both epsilon exploration and churn (epsilon=0 with target-network acting) (ablation) | DoubleDQN · Atari 15 games | return | Hurt |
| acting network synced from online network more often than every ~1 000 gradient updates (ablation) | DoubleDQN with decoupled acting network · Atari | return | Improved |
| expose full Atari action set (redundant actions) (ablation) | DoubleDQN · Atari | policy churn magnitude | No clear effect |
| add Advantage Learning to enlarge action gaps (ablation) | DQN · Atari | policy churn | Improved |
| AL-DQN trained with epsilon=0 (ablation) | AL-DQN · Atari | training success | No clear effect |
| stationary data (forked tandem) instead of on-policy data (ablation) | DoubleDQN · Atari | policy churn | No clear effect |
| Monte-Carlo return targets instead of TD targets (ablation) | forked tandem DoubleDQN · Atari | policy churn | No clear effect |
| switch from deep network to linear function approximation (ablation) | Q-learning agent · Atari | policy change | Improved |
| switch from DoubleDQN to R2D2 setup (recurrent, Adam, large batch, 192 actors) (ablation) | agent · Atari | mean policy change per update (%): 9 -> 6 | No clear effect |
| wider network (200 vs 50 units) (ablation) | DQN-like and 3-layer MLP Q-learning · Catch | policy change | Mixed |
Closest papers by meaning
- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.92)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.86)
- Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · similarity 0.86)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.86)
- Reward Inflation: A Healthy Stimulus for Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
