Research map / Reinforcement learning / Policy gradient and actor-critic
Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes value and policy churn in generalized policy iteration, identifies a chain effect of churn, and proposes CHAIN, a plug-in regularizer on a separate reference batch that reduces churn and improves DRL performance and scaling.
Problem
Neural network function approximation in deep RL makes value and policy outputs change unexpectedly on states outside the current training batch (churn), and how this churn compounds and harms learning in different DRL settings is under-explored.
Approach
Formalize value and policy churn within generalized policy iteration, show they form a chain effect that biases parameter updates, and reduce it by adding a regularizer that penalizes output changes on a separate reference batch relative to the previous network (CHAIN), with an automatic coefficient adjustment based on a target relative loss scale.
Key results
- CHAIN PPO-style regularization improved final return of PPO scaled to 8x wider networks on Ant-v4 at 10M steps (5592.94 vs 4235.63).
- CHAIN IQL (PCR) improved AM-umaze-v2 from 77.00 to 84.44 and CHAIN AWAC (PCR) improved pen-human-v1 from 81.12 to 99.72.
- In DoubleDQN on MinAtar, value churn and greedy action deviation (over 20% of states) accumulate with training, and CHAIN reduces both significantly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add policy churn reduction (ablation) | IQL · Antmaze D4RL tasks | normalized score: 77.00 $\pm$ 5.52 -> 84.44 $\pm$ 3.19; 54.25 $\pm$ 5.54 -> 62.50 $\pm$ 3.75; 65.75 $\pm$ 11.71 -> 72.50 $\pm$ 2.92; 73.75 | Mixed |
| add value churn reduction (ablation) | IQL · Antmaze D4RL tasks | normalized score: 77.00 $\pm$ 5.52 -> 83.33 $\pm$ 2.72; 54.25 $\pm$ 5.54 -> 71.67 $\pm$ 7.23; 65.75 $\pm$ 11.71 -> 70.00 $\pm$ 3.33; 73.75 | Mixed |
| add policy churn reduction (ablation) | AWAC · Adroit pen-human-v1 and pen-cloned-v1 | normalized score: 81.12 $\pm$ 13.47 -> 99.72 $\pm$ 2.04; 89.56 $\pm$ 15.57 -> 95.49 $\pm$ 2.34 | Helped |
| add value churn reduction (ablation) | AWAC · Adroit pen-human-v1 and pen-cloned-v1 | normalized score: 81.12 $\pm$ 13.47 -> 97.37 $\pm$ 3.51; 89.56 $\pm$ 15.57 -> 96.66 $\pm$ 2.54 | Helped |
| add CHAIN churn reduction | PPO (4x wider) · Ant, Human., and Walker2d (10M) | episode return: 3013.95 $\pm$ 223.77 -> \textbf{4916.66} $\pm$ 109.01; 2998.18 $\pm$ 237.95 -> \textbf{4830.58} $\pm$ 231.42; 3795.05 $\ | Helped |
| add CHAIN churn reduction | PPO (4x deeper) · Ant, Human., and Walker2d (10M) | episode return: 2777.62 $\pm$ 136.78 -> \textbf{3760.46} $\pm$ 158.29; 3489.47 $\pm$ 166.28 -> \textbf{4090.80} $\pm$ 187.01; 2845.68 $\ | Helped |
| add CHAIN churn reduction | PPO (8x wider) · Ant, Human., and Walker2d (10M) | episode return: 4235.63 $\pm$ 209.68 -> \textbf{5592.94} $\pm$ 205.23; 3198.01 $\pm$ 344.71 -> \textbf{5246.15} $\pm$ 130.73; 4335.17 $\ | Helped |
| add CHAIN churn reduction | PPO (8x deeper) · Ant, Human., and Walker2d (10M) | episode return: 3364.46 $\pm$ 173.25 -> \textbf{4278.76} $\pm$ 122.61; 2780.67 $\pm$ 304.10 -> \textbf{4434.48} $\pm$ 144.92; 3057.89 $\ | Helped |
| add value churn reduction | DoubleDQN · MinAtar | episode return | Helped |
| add automatic adjustment of the regularization coefficient (ablation) | CHAIN DoubleDQN · MinAtar | episode return | No clear effect |
| add policy churn reduction | PPO · MuJoCo and DMC | episode return | Helped |
| add policy churn reduction (ablation) | TD3 and SAC · MuJoCo | episode return | Mixed |
Surprising findings
- Policy value deviation from policy churn was positive overall, suggesting an implicit optimization effect on reference states, though the authors caution it could be delusional.
- Scaling PPO degrades severely with network size, and CHAIN only achieves sub-linear scaling rather than fully resolving the scaling issue.
Limitations
- No formal theoretical analysis of the chain effect under concrete assumptions; analysis relies on NTK approximations.
- Regularization coefficients are domain-dependent; auto-adjustment with a target relative loss scale does not transfer hyperparameters across domains.
- Choosing among policy, value, or dual churn reduction for actor-critic methods is unresolved.
Experimental setup
Datasets: MinAtar OpenAI MuJoCo DeepMind Control Suite D4RL
Benchmarks: MinAtar MuJoCo DeepMind Control Suite D4RL Antmaze D4RL Adroit
Metrics: episode return sample efficiency value churn policy churn greedy action deviation final score
Baselines: DoubleDQN PPO TD3 SAC IQL AWAC
Builds on: DoubleDQN PPO TD3 SAC IQL
Closest papers by meaning
- Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn (ICML 2025 · similarity 0.89)
- Efficient Reinforcement Learning by Discovering Neural Pathways (NeurIPS 2024 · similarity 0.88)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.88)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.87)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.86)
- Neural Network Approximation for Pessimistic Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.