Research map / Reinforcement learning / Visual and representation RL
Neuroplastic Expansion in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neuroplastic Expansion (NE), which grows a sparse TD3/DrQ network from a small initial size toward full capacity using gradient-guided growth, dormant-neuron pruning, and experience review to mitigate plasticity loss in deep RL.
Problem
Deep RL agents lose plasticity during training (dormant neurons, declining activated neuron ratio), hurting learning in non-stationary settings; existing reset-based and implementation-level fixes are unstable or lack generality.
Approach
Start with a small sparse actor-critic (about 25% capacity) and dynamically grow it to full size using top-k gradient-magnitude candidate connections, prune fully dormant neurons each growth round, and periodically replay early buffer experience when the activated-neuron slope flattens to preserve stability.
Key results
- On MuJoCo long-term training, NE outperforms Reset, ReDo, LN and Plasticity Injection baselines in four tasks while maintaining higher activated neuron ratio than LN.
- In continual adaptation, NE-TD3 significantly outperforms vanilla TD3 and performs comparably to Reset in most tasks, while clearly outperforming Reset on Humanoid.
- With DrQ on four DMC image-based tasks, NE outperforms all baselines, though it does not reach Reset's peak activated neuron ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add experience review (ablation) | NE · Ant, end of training | return: 7235.14± 493.83 -> 7749.25± 127.73 | Helped |
| add experience review (ablation) | NE · Reacher Hard, end of training | return: 419.25± 65.73 -> 475.46±34.11 | Helped |
| add experience review (ablation) | NE · Cartpole Swingup Sparse, end of training | return: 402.88± 71.23 -> 424.76± 47.68 | Helped |
| add experience review (ablation) | NE · Hammer sparse | success rate: 0.47± 0.21 -> 0.54± 0.13 | Helped |
| add experience review (ablation) | NE · Sweep Into sparse | success rate: 0.52± 0.16 -> 0.61± 0.09 | Helped |
| add dormant-neuron pruning (ablation) | NE · Cartpole Swingup Sparse and Walker Walk | return | Helped |
| vary initial sparsity (ablation) | NE · Ant and DMC Reacher Hard | return | Mixed |
| vary growth rate (ablation) | NE · Ant and DMC Reacher Hard | return | Mixed |
Surprising findings
- Naive random topology growth alone already improves TD3 performance and reduces neuron dormancy.
- Without experience review, NE shows high variance late in training because agents forget early skills such as standing in HalfCheetah.
Limitations
- Growth increases computation cost, and the gradient-based candidate selection introduces errors that pruning must correct.
- Hyperparameters (initial sparsity, growth rate, pruning bound, ER threshold) are task-dependent and tuned empirically.
- Performance is not uniformly best: PI achieves higher neuron activation, and NE does not reach Reset's peak activation on image tasks.
Experimental setup
Datasets: OpenAI Gym MuJoCo (Hopper, Walker2D, HalfCheetah, Ant, Humanoid) DeepMind Control Suite (Reacher Hard, Reacher Easy, Walker Walk, Cartpole Swingup Sparse)
Benchmarks: OpenAI MuJoCo DeepMind Control Suite
Metrics: episodic return activated neuron ratio standard deviation across seeds
Baselines: Reset ReDo Layer Normalization (LN-TD3) Plasticity Injection vanilla TD3 DrQ
Builds on: TD3 DrQ ReDo Plasticity Injection sparse training (RigL-style gradient growth)
Closest papers by meaning
- Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks (NeurIPS 2025 · similarity 0.89)
- Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning (ICML 2025 · similarity 0.87)
- Adaptive Rational Activations to Boost Deep Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks (ICML 2024 · similarity 0.85)
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.85)
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.