Research map / Architectures and efficiency / Pruning and efficient tuning

Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Sparse training from scratch · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that static one-shot random pruning (ER-allocated sparsity) of SimBa actor/critic networks lets DRL agents scale beyond dense SimBa limits, with better parameter efficiency, less plasticity loss, and less gradient interference, and extends gains to visual RL and streaming RL.

Problem

Scaling deep RL networks degrades performance due to optimization pathologies (plasticity loss, capacity collapse, parameter growth, gradient interference), and even the state-of-the-art SimBa architecture stops improving and degrades when made larger.

Approach

Introduce static weight-level sparsity via one-shot random pruning at initialization (with Erdős–Rényi layer-wise ratios) on top of SimBa, so that larger sparse networks at fixed learnable-parameter budgets outperform dense ones without resets or algorithm changes; diagnostics attribute gains to higher Srank, lower dormant ratio, lower parameter norm, higher simplicity bias, and reduced gradient i

Key results

• On four hardest DMC tasks, sparse large SimBa networks (~109M params) keep improving with sparsity while dense networks degrade with scale; sparse networks reach up to 40% higher returns than dense counterparts on Humanoid Walk and Dog Run.
• Large dense critics show capacity collapse (Srank drop) and rising dormant ratios, while large sparse networks avoid Reset dependence; Reset helps dense large nets but gives no benefit to sparse ones.
• In visual RL (DrQ-v2), combining 0.8 sparsity with 4x wider critic significantly boosts returns over dense wider critics, and avoids dense-net plasticity deterioration.

Measured effects reported in the paper

Surprising findings

• Reset operations, which substantially help large dense networks, provide no benefit and can slightly harm large sparse networks.
• Large sparse networks can have similar or lower parameter norms than small dense networks with equal learnable parameters.

Limitations

• Only random one-shot pruning studied; structured or targeted sparsity not explored.
• Atari-100k low-data regime may not reveal scaling benefits, per the authors.
• Theoretical understanding of why sparsity helps is left open.

Experimental setup

Datasets: DeepMind Control Suite (DMC Hard, 6 tasks) DMC image-based tasks MuJoCo Ant-v4 and Walker2d-v4 Atari-100k

Benchmarks: DMC Hard (6 tasks) DMC image-based (visual RL) MuJoCo Ant-v4 / Walker2d-v4 (streaming RL) Atari-100k

Metrics: episode return stable rank (Srank) dormant ratio gradient norm parameter norm simplicity bias score gradient covariance interference fraction of active units

Baselines: dense SimBa Reset SparseInit Soft MoE value networks Neuroplastic Expansion gradual magnitude pruning

Builds on: SimBa BRO DrQ-v2 Stream AC(λ) Sparse Training (ER initialization)

Closest papers by meaning

• Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• In value-based deep reinforcement learning, a pruned network is a good network (ICML 2024 · similarity 0.88)
• Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.88)
• Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
• 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities (NeurIPS 2025 · similarity 0.87)