Research map / Continual learning / Online and replay-based
Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-CHAIN, which penalizes out-of-batch network output churn during continual RL to prevent NTK rank collapse and mitigate plasticity loss.
Problem
Loss of plasticity in continual reinforcement learning, where agents gradually lose the ability to adapt to new tasks; the mechanism linking it to network output churn and NTK rank collapse is not well understood.
Approach
Connects plasticity loss to churn (output change on out-of-batch reference data) via the empirical NTK, and adds a churn-reduction loss on a reference batch disjoint from the training batch to decorrelate gradients and adaptively rescale RL gradient steps.
Key results
- On continual ProcGen (16 environments), C-CHAIN achieves an aggregate score of 101.792 versus 77.289 for the second-best (TRAC) and 55.049 for Vanilla PPO.
- On continual DMC, C-CHAIN improves over Vanilla PPO in all three settings, e.g., C-Walker 472.828 vs 305.199.
- C-CHAIN maintains a higher, more stable NTK approximate rank and lower off-diagonal NTK values than Vanilla PPO on continual ProcGen Starpilot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reduce churn during continual PPO training | PPO · Continual Gym Control | Agg. Score: -243.985 -> -221.22 | Helped |
| Reduce churn during continual PPO training | PPO · Continual ProcGen | Agg. Score: 77.289 -> 101.792 | Helped |
| Reduce churn during continual PPO training | PPO · Continual DMC: C-Walker, C-Quadruped, and C-Dog | episode return: 395.971 $\pm$ 8.116 -> 472.828 \pm 17.865; 250.153 $\pm$ 26.385 -> 314.510 \pm 44.321; 137.080 $\pm$ 3.133 -> 174.098 \p | Helped |
| Reduce churn during continual DoubleDQN training | DoubleDQN · C-MinAtar | episode return: 22.044 $\pm$ 0.733 -> 29.513 \pm 0.682 | Helped |
| Reduce churn during continual learning | Vanilla agent · RandomLabel-MNIST and Permuted-MNIST | accuracy: 0.1501 $\pm$ 0.0030 -> 0.2482 $\pm$ 0.0141; 0.6430 $\pm$ 0.0029 -> 0.6797 $\pm$ 0.0042 | Helped |
| Reduce churn instead of using L2 Init | Continual learning agent · RandomLabel-MNIST | accuracy: 0.8607 $\pm$ 0.0021 -> 0.2482 $\pm$ 0.0141 | Hurt |
| Reduce churn instead of using Weight Clipping | Continual learning agent · Permuted-MNIST | accuracy: 0.8281 $\pm$ 0.0001 -> 0.6797 $\pm$ 0.0042 | Hurt |
| Use the orthogonal churn-reduction gradient component only (ablation) | C-CHAIN · ProcGen Starpilot and Chaser | episode return | Helped |
| Use the projective churn-reduction gradient component only (ablation) | C-CHAIN · ProcGen Starpilot and Chaser | episode return | Helped |
| Use the orthogonal rather than the projective churn-reduction component (ablation) | C-CHAIN · ProcGen Starpilot and Chaser | episode return | Helped |
| Increase reference batch size to 2x, 4x, or 8x the training batch size (ablation) | C-CHAIN · Continual Gym Control | episode return | Mixed |
Surprising findings
- On continual supervised MNIST settings, C-CHAIN underperforms L2 Init and Weight Clipping, which the authors attribute to the simplicity of those tasks.
- Naive network resetting (Oracle) is outperformed by C-CHAIN and TRAC on some ProcGen environments, indicating resets are not sufficient to address plasticity loss in continual RL.
Limitations
- C-CHAIN is less effective on continual supervised learning MNIST benchmarks than L2 Init and Weight Clipping.
- Reducing churn can slow exploration and early learning, e.g., poor early performance on continual MountainCar.
- Authors note that C-CHAIN does not fully avoid near-zero minimum scores on sparse-reward ProcGen environments due to PPO's limited exploration.
Experimental setup
Datasets: CartPole-v1 Acrobot-v1 LunarLander-v2 MountainCar-v0 ProcGen DeepMind Control Suite MinAtar RandomLabel-MNIST Permuted-MNIST
Benchmarks: Gym Control (continual) ProcGen (16 environments, continual) DMC (continual Walker, Quadruped, Dog) MinAtar (continual) RandomLabel-MNIST Permuted-MNIST
Metrics: average performance (episode return, AUC-style) mean median IQM optimality gap approximate rank of NTK off-diagonal NTK sum
Baselines: TRAC Weight Clipping L2 Init ReDo LayerNorm AdamRel Oracle PPO Vanilla PPO
Builds on: PPO TRAC L2 Init Neural Tangent Kernel Double DQN
Closest papers by meaning
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.89)
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.88)
- Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective (ICML 2025 · similarity 0.87)
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR 2024 · similarity 0.87)
- Temporal-Difference Variational Continual Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.