Research map / Reinforcement learning / Policy gradient and actor-critic
In value-based deep reinforcement learning, a pruned network is a good network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that gradual magnitude pruning (95% sparsity) of wide ResNet value-based deep RL agents (DQN, Rainbow, IQN, DrQ(ε), CQL) improves Atari performance over dense baselines while using a small fraction of parameters.
Problem
Deep RL agents under-utilize their network parameters and suffer from plasticity loss and dormant neurons as networks scale; it is unclear whether sparse training can make value-based agents parameter-efficient.
Approach
Apply gradual magnitude pruning (cubic schedule, 20%-80% of training, target 95% sparsity) to value-based RL agents with widened Impala ResNets, so that the pruned wide network outperforms dense baselines and its performance grows with width, unlike dense networks.
Key results
- Pruning DQN with a 3x-wide Impala ResNet at 95% sparsity yields close to 60% (DQN) and 50% (Rainbow) IQM improvement over the original unpruned, unscaled architectures.
- Pruned networks' performance increases monotonically with width while dense networks' performance decreases with width.
- Pruning outperforms RigL, lottery ticket, weight decay, L2, reset and ReDo baselines across widths.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply gradual magnitude pruning to a width-3 Impala ResNet at 95% sparsity | DQN · Atari, 15 games | human-normalized IQM performance: close to a 60% | Helped |
| Apply gradual magnitude pruning to a width-3 Impala ResNet at 95% sparsity | Rainbow · Atari, 15 games | human-normalized IQM performance: 50% | Helped |
| Apply gradual magnitude pruning at 90% or 95% sparsity to a width-3 Impala ResNet | DQN · Atari, 15 games | performance: 33% | Helped |
| Increase width of the initial network while using gradual magnitude pruning | DQN and Rainbow · Atari, 15 games | human-normalized IQM performance | Mixed |
| Apply gradual magnitude pruning in the 100k data regime | Atari 100k agents · Atari 100k benchmark | performance | No clear effect |
| Apply gradual magnitude pruning during 40M-frame training | DrQ(ε) and DER · Atari, 40M frames | performance | Helped |
| Apply gradual magnitude pruning | CQL and CQL+C51 · Atari offline RL, 17 games | interquantile mean performance | Helped |
| Apply gradual magnitude pruning to width-5 networks | SAC · Walker2d-v2 and Ant-v2 | return | Helped |
| Apply gradual magnitude pruning to width-5 networks | SAC · Three remaining MuJoCo environments | return | No clear effect |
| Use the original CNN architecture rather than the Impala ResNet (ablation) | DQN with gradual magnitude pruning · Atari | performance | No clear effect |
| Replace gradual pruning with a fixed final mask and original initialization (ablation) | DQN · Atari, 40M frames | performance | Hurt |
| Use RigL instead of gradual magnitude pruning (ablation) | DQN · Atari | performance | Hurt |
Surprising findings
- Dense networks' performance decreases monotonically as width increases, whereas pruned networks improve with width.
- Pruning gave no gains on the low-data Atari 100k regime, and no gains on SAC/PPO MuJoCo in most environments.
Limitations
- Most experiments use 40M frames rather than the standard 200M, and the pruning schedule must be adjusted for different training lengths.
- No gains on Atari 100k low-data regime and limited gains on SAC/PPO MuJoCo in most environments.
- Experiments run on NVIDIA P100 GPUs; the paper notes the pruned sparse networks do not yet benefit from sparse hardware acceleration.
Experimental setup
Datasets: Arcade Learning Environment (15-game subset) Atari 100k 26-game set Atari 60-game set 17-game offline dataset from DQN replay (5%) MuJoCo
Benchmarks: Arcade Learning Environment (15 games) Atari 2600 full 60-game suite Atari 100k MuJoCo (SAC, PPO)
Metrics: human-normalized interquantile mean (IQM) return srank dormant neuron fraction Q-variance parameter norm
Baselines: dense DQN dense Rainbow lottery ticket RigL weight decay L2 normalization network reset ReDo
Builds on: gradual magnitude pruning (Zhu & Gupta) Impala ResNet DQN Rainbow JaxPruner Dopamine
Closest papers by meaning
- Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning (ICML 2025 · similarity 0.88)
- Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training (NeurIPS 2024 · similarity 0.87)
- Learning the Target Network in Function Space (ICML 2024 · similarity 0.86)
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (NeurIPS 2025 · similarity 0.86)
- Mixtures of Experts Unlock Parameter Scaling for Deep RL (ICML 2024 · similarity 0.85)
- Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.