AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

The Phenomenon of Policy Churn

Tom Schaul, Andre Barreto, John Quan, Georg Ostrovski

NeurIPS 2022 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
act with epsilon=0 instead of decaying epsilon-greedy (ablation)DoubleDQN · Atari 15 gamesreturnNo clear effect
act with the target network (updated every 120 000 frames) to remove churn from behaviour (ablation)DoubleDQN with epsilon-greedy · Atari 15 gamesreturnHurt
remove both epsilon exploration and churn (epsilon=0 with target-network acting) (ablation)DoubleDQN · Atari 15 gamesreturnHurt
acting network synced from online network more often than every ~1 000 gradient updates (ablation)DoubleDQN with decoupled acting network · AtarireturnImproved
expose full Atari action set (redundant actions) (ablation)DoubleDQN · Ataripolicy churn magnitudeNo clear effect
add Advantage Learning to enlarge action gaps (ablation)DQN · Ataripolicy churnImproved
AL-DQN trained with epsilon=0 (ablation)AL-DQN · Ataritraining successNo clear effect
stationary data (forked tandem) instead of on-policy data (ablation)DoubleDQN · Ataripolicy churnNo clear effect
Monte-Carlo return targets instead of TD targets (ablation)forked tandem DoubleDQN · Ataripolicy churnNo clear effect
switch from deep network to linear function approximation (ablation)Q-learning agent · Ataripolicy changeImproved
switch from DoubleDQN to R2D2 setup (recurrent, Adam, large batch, 192 actors) (ablation)agent · Atarimean policy change per update (%): 9 -> 6No clear effect
wider network (200 vs 50 units) (ablation)DQN-like and 3-layer MLP Q-learning · Catchpolicy changeMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy