Research map / Reinforcement learning / Visual and representation RL
A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies plasticity loss in on-policy PPO under permute, window, and expand distribution shifts across gridworld, CoinRun, and Montezuma's Revenge, and benchmarks existing mitigation methods.
Problem
Plasticity loss (a degraded ability of a network trained online to fit new data after distribution shift) has been studied in supervised and off-policy RL but little in on-policy deep RL; it is unclear which mitigation methods work there.
Approach
Introduces three distribution-shift protocols (permute, window, expand) to induce plasticity loss in on-policy PPO and systematically compares intermittent and continuous interventions, finding that continual regularizers toward initialization (soft shrink+perturb, regenerative regularization, LayerNorm combined with them) work best while CReLU and plasticity injection do not transfer.
Key results
- Under permute, window, and expand shifts, warm-started PPO shows clear training-reward degradation relative to reset-all in gridworld, indicating plasticity loss in on-policy RL.
- Regenerative regularization and soft shrink+perturb mitigate plasticity loss across all three gridworld shift conditions, and the most effective methods (LayerNorm, regenerative regularization, soft shrink+perturb) also work best on CoinRun.
- On Montezuma's Revenge with RND, soft shrink+perturb or regenerative regularization achieves higher reward than a standard RND agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CReLU instead of ReLU | PPO on Gridworld · Permute shift | normalized reward | Hurt |
| Apply plasticity injection to final layers | PPO on Gridworld · Permute, Window, and Expand shifts | normalized reward | Hurt |
| Reset the final layer periodically | PPO on Gridworld · Permute, Window, and Expand shifts | normalized reward | No clear effect |
| Use soft shrink+perturb each optimization step | PPO on Gridworld · Permute, Window, and Expand shifts | training plasticity loss and warm-start performance | Helped |
| Regularize weights toward their initialization | PPO on Gridworld · Permute, Window, and Expand shifts | training plasticity loss and warm-start performance | Helped |
| Use ReDo neuron resets | PPO on Gridworld · Expand shift | normalized reward | Hurt |
| Add LayerNorm | PPO on CoinRun · all three shifts | training plasticity loss | Helped |
| Add LayerNorm | PPO on CoinRun · Permute shift test data | test performance | Hurt |
| Combine LayerNorm with soft shrink+perturb or regenerative regularization | PPO on CoinRun · distribution shifts | training and generalization performance | Helped |
| Use soft shrink+perturb or regenerative regularization | RND agent on Montezuma's Revenge · Montezuma's Revenge | reward | Helped |
| Change from warm-start to reset-all between rounds | PPO on Gridworld · Permute and Window shifts | mean episodic reward | Helped |
| Change from warm-start to reset-all between rounds | PPO on Gridworld · Expand shift, test distribution | generalization performance | Helped |
Surprising findings
- CReLU and plasticity injection, effective in off-policy RL, fail to resolve plasticity loss in the on-policy setting; plasticity injection underperforms even the warm-start baseline in all three gridworld conditions.
- LayerNorm resolves training plasticity loss but significantly underperforms warm-start on test data in the permute condition.
Limitations
- Gridworld and CoinRun experiments use five seeds each and a small set of environments, limiting generality.
- Correlation and GLM analyses are non-causal, as the authors acknowledge.
- Montezuma's Revenge evaluated only with soft shrink+perturb and regenerative regularization due to compute constraints.
Experimental setup
Datasets: NeuroNav gridworld CoinRun (ProcGen) Montezuma's Revenge (ALE) ProcGen Jumper ProcGen Fruitbot
Benchmarks: NeuroNav gridworld CoinRun Montezuma's Revenge Jumper Fruitbot
Metrics: normalized mean episodic reward train reward test reward statistical t-tests
Baselines: warm-start reset-all reset-final CReLU plasticity injection ReDo L2 norm shrink+perturb LayerNorm
Builds on: PPO shrink+perturb ReDo plasticity injection regenerative regularization
Closest papers by meaning
- Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn (ICML 2025 · similarity 0.91)
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.90)
- Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning (ICML 2025 · similarity 0.89)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.87)
- Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound (NeurIPS 2025 · similarity 0.87)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.