Research map / Reinforcement learning / Policy gradient and actor-critic

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning

Read the paper Check your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Sustainable Backup Propagation (SBP), a cyclic neuron-reset method with inner distillation, instantiated as Plastic PPO (P3O) using an alpha-weighted double KL loss to counter plasticity loss in deep RL.

Problem

Deep RL agents lose plasticity over training due to primacy bias and dead/dormant neurons; existing resets either are computationally costly (large replay buffers) or damage performance without knowledge recovery.

Approach

Introduce neuron regeneration: periodically reset a rotating subset of neurons (cycle reset) and recover their knowledge via distillation from a pre-reset copy of the policy to the non-reset neurons, using an alpha-weighted combination of forward and reverse KL divergence.

Key results

P3O reaches 6237 average return on Humanoid vs 1163 for PPO (436% improvement), averaged over 25 seeds.
P3O improves over PPO on Walker (6788 vs 6270) and Ant (5522 vs 3354) with non-overlapping confidence intervals on Ant and Humanoid.
P3O maintains smaller actor weight norms and higher gradient norms than PPO and reset baselines, per the analysis in Figures 4 and 5.

Measured effects reported in the paper

What was changed	Where	Measured	Effect
Integrate SBP neuron regeneration and α-DKL distillation into PPO	PPO · Hopper	average return: 3719 -> 3765	Helped
Integrate SBP neuron regeneration and α-DKL distillation into PPO	PPO · HumanoidStandup	average return: 135k -> 144k	Helped
Integrate SBP neuron regeneration and α-DKL distillation into PPO	PPO · Walker	average return: 6270 -> 6788	Helped
Integrate SBP neuron regeneration and α-DKL distillation into PPO	PPO · Ant	average return: 3,354 -> 5522	Helped
Integrate SBP neuron regeneration and α-DKL distillation into PPO	PPO · HalfCheetah	average return: 3626 -> 6865	Helped
Integrate SBP neuron regeneration and α-DKL distillation into PPO	PPO · Humanoid	average return: 1163 -> 6237	Helped
Add inner distillation after cycle reset (ablation)	P3O · MuJoCo environments	average return	Helped
Increase the number of PPO actor training epochs per batch (ablation)	PPO · Humanoid, Hopper, Walker2d	average return	Hurt
Increase recovery training epochs without inner distillation (ablation)	P3O · MuJoCo environments	average return	No clear effect

Surprising findings

On Hopper, P3O's gain over PPO is only 1% (3765 vs 3719), with overlapping confidence intervals.
PPO+CBP outperforms P3O on Walker-type comparisons? No: PPO+CBP scores below PPO on Walker (5284 vs 6270), showing naive selective resets can hurt.

Limitations

Distillation adds training epochs and computational overhead.
Reset scheduling is not optimized and lacks theoretical analysis of neuron regeneration.
Evaluation is limited to PPO and state-based continuous control tasks.

Experimental setup

Datasets: MuJoCo DeepMind Control Suite Cycle Friction (MuJoCo variant)

Benchmarks: MuJoCo (Hopper, HumanoidStandup, Walker, Ant, HalfCheetah, Humanoid) DeepMind Control Suite (Dog Walk, Quadruped Walk, Quadruped Run, Hopper Hop) Cycle Friction (Ant-CF, Walker-CF, Hopper-CF)

Metrics: average return 95% confidence interval actor weight L1 norm gradient norm

Baselines: PPO PPO+CBP PPO+ReDo PPO+Cycle

Builds on: PPO CBP ReDo policy distillation

Closest papers by meaning

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control (ICML 2025 · similarity 0.87)
The Dormant Neuron Phenomenon in Multi-Agent Reinforcement Learning Value Factorization (NeurIPS 2024 · similarity 0.87)
Self-Normalized Resets for Plasticity in Continual Learning (ICLR 2025 · similarity 0.87)
Neuroplastic Expansion in Deep Reinforcement Learning (ICLR 2025 · similarity 0.87)
Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.86)

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft