Research map / Reinforcement learning / Visual and representation RL
Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GraMa, a gradient-magnitude-based neuron activity metric, and ReGraMa, a reset scheme guided by it, to restore learning capacity of dormant neurons in advanced deep RL agents (BRO-net, DACER diffusion policy, SAC variants).
Problem
Activation-based dormant neuron metrics (tau-dormant ratio / ReDo) lose statistical power in advanced deep RL architectures with residual connections, diffusion policies, normalization, and diverse activations, so neuron resets misidentify inactive neurons.
Approach
Shift the neuron health statistic from activation magnitudes to gradient magnitudes, measuring a neuron's learning capacity (ability to adapt via gradient updates) rather than its expressive output, and reset neurons flagged by this metric.
Key results
- On BRO-net DMC tasks, ReGraMa achieves normalized performance of 1.21 on Humanoid stand and 1.08 on Dog walk versus 1.17 and 0.94 for ReDo (relative to vanilla BRO-net).
- ReGraMa keeps the proportion of inactive neurons consistently lower than ReDo in DACER on Ant and Walker2d.
- With SAC using Tanh or Sigmoid activations, ReDo degrades, sometimes falling below vanilla SAC, while ReGraMa remains robust.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| guide neuron resetting with GraMa (ReGraMa) instead of vanilla BRO-net resetting | BRO-net · DMC Humanoid Stand | normalized performance: 1.0 -> 1.21 \pm 0.03 | Helped |
| guide neuron resetting with GraMa (ReGraMa) instead of vanilla BRO-net resetting | BRO-net · DMC Humanoid Run | normalized performance: 1.0 -> 1.16 \pm 0.08 | Helped |
| guide neuron resetting with GraMa (ReGraMa) instead of vanilla BRO-net resetting | BRO-net · DMC Dog Stand | normalized performance: 1.0 -> 1.12 \pm 0.08 | Helped |
| guide neuron resetting with GraMa (ReGraMa) instead of vanilla BRO-net resetting | BRO-net · DMC Dog Walk | normalized performance: 1.0 -> 1.08 \pm 0.04 | Helped |
| add L2 initialization (ablation) | ReGraMa with DrQv2 · DMC Quadruped Run | performance: 706 \pm 127.25 -> 742.36 \pm 127.31 | Helped |
| add weight decay (ablation) | ReGraMa with DrQv2 · DMC Quadruped Run | performance: 706 \pm 127.25 -> 751.31 \pm 94.26 | Helped |
| add both L2 initialization and weight decay (ablation) | ReGraMa with DrQv2 · DMC Quadruped Run | performance: 706 \pm 127.25 -> 739.42 \pm 104.58 | No clear effect |
Surprising findings
- ReDo's activation-based metric fails to identify neurons losing learning capacity in residual BRO-net and diffusion-based DACER policies, sometimes underperforming vanilla agents.
- With Tanh activations, the proportion of inactive neurons under ReGraMa still gradually increases during training, which the authors leave unexplained.
Limitations
- Evaluation limited to three representative neural architectures due to computational constraints.
- Mechanism behind persistent inactive neurons under Tanh activation with ReGraMa is not explained.
- Does not extend to large-scale transformer policies, multi-task or continual RL in current work.
Experimental setup
Datasets: MuJoCo DeepMind Control Suite
Benchmarks: DeepMind Control Suite MuJoCo
Metrics: episode return normalized score proportion of inactive neurons
Baselines: ReDo BRO-net DACER ReBorn S&P CBP
Builds on: ReDo (tau-dormant neuron ratio) BRO-net DACER SAC
Closest papers by meaning
- The Dormant Neuron Phenomenon in Multi-Agent Reinforcement Learning Value Factorization (NeurIPS 2024 · similarity 0.87)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.86)
- Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization (ICLR 2024 · similarity 0.86)
- Adaptive Rational Activations to Boost Deep Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.