Research map / Architectures and efficiency / CNN design
Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that scaling pixel-based deep RL networks hurts mainly through the encoder-to-dense bottleneck, and proposes Global Average Pooling on encoder outputs as a simple fix that matches or beats SoftMoE and pruning.
Problem
Scaling the width of pixel-based deep RL networks (e.g., Rainbow on Atari) typically degrades performance, and recent fixes (SoftMoE tokenization, sparse training) are complex and costly; the underlying cause was unclear.
Approach
Identifies the encoder-to-dense connection (the bottleneck) as the main limit on scaling and argues prior methods work by structuring/sparsifying it; proposes replacing flattening with global average pooling over each feature map so the dense layers get a C-dimensional input.
Key results
- GAP improves performance of scaled networks across scales, outperforming baselines including pruning, RigL, weight decay and layer norm, and achieving performance comparable to SoftMoE-1 at 100M steps.
- Scaling only the bottleneck (first dense layer after the encoder) accounts for most of the performance degradation seen when scaling the full network.
- Sparsifying only the dense layers psi yields better performance than sparsifying both encoder and dense layers across sparse methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Global Average Pooling (GAP) applied to encoder output before dense layers | Rainbow with scaled width (x4) on Atari 20 games · Atari 20-game subset | IQM score | Helped |
| Global Average Pooling (GAP) applied to encoder output before dense layers | Rainbow on full 60-game Atari suite · Atari 60 games | IQM score | Helped |
| Global Average Pooling (GAP) applied to encoder output before dense layers | Rainbow with high replay ratio (sample-efficient regime) · Atari 20 games, replay ratio up to default 0.25 and higher | IQM score | Helped |
| Global Average Pooling (GAP) applied to encoder output before dense layers | CNN architecture (Nature-style) encoder with scaled dense layers · Atari 20 games | IQM score | Helped |
| Global Average Pooling (GAP) applied to encoder output | Rainbow on Procgen, DER on Atari100K, SAC on DMC · Procgen, Atari100K, DMC | performance | Helped |
| Global Average Pooling vs. global max pooling (ablation) | Rainbow on 20 Atari games · Atari 20 games | performance | Helped |
| Sparsifying only the bottleneck (psi) instead of both phi and psi (ablation) | Gradual pruning, RigL, static sparsity on scaled Rainbow · Atari 20 games, 90% sparsity | IQM score | Helped |
| Scaling network width of the whole network (phi and psi) without pooling | Scaled baseline Rainbow (x4) · Atari 20 games | IQM score | Hurt |
| Increasing depth of the dense layers psi in the baseline (ablation) | Rainbow baseline · Atari 20 games | performance | Hurt |
| Adding GAP when increasing depth of psi (ablation) | Rainbow with deeper dense layers · Atari 20 games | performance | Helped |
| Adding GAP to a encoder-width-scaled network (psi unscaled) (ablation) | Rainbow with scaled phi · Atari 20 games | performance | Helped |
| Increasing encoder depth by adding ResNet blocks to scaled bottleneck (ablation) | Rainbow with scaled psi · Atari 20 games | performance | Helped |
Surprising findings
- Scaled dense layers show much higher dormant-neuron fractions than the encoder, while the encoder has low dormancy.
- GAP is more computationally efficient than SoftMoE and sparse methods, since sparsity is simulated via masking which keeps runtime high.
Limitations
- Focused on pixel-based environments with a clear encoder-dense bottleneck; unclear whether findings transfer to non-pixel or architectures without a clear bottleneck.
- GAP is a more aggressive compression that may discard spatial structure compared with finer tokenization in SoftMoE.
Experimental setup
Datasets: Arcade Learning Environment (60 games; 20-game subset) Atari100K (26 games) Procgen DeepMind Control Suite
Benchmarks: Arcade Learning Environment 20-game subset Arcade Learning Environment 60-game suite Atari100K Procgen DeepMind Control Suite
Metrics: interquartile mean (IQM) median mean GPU hours
Baselines: SoftMoE-1 gradual pruning RigL static sparsity weight decay layer normalization DrQ DER
Builds on: Rainbow Impala SoftMoE RigL gradual pruning
Closest papers by meaning
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning (ICML 2025 · similarity 0.87)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Mixtures of Experts Unlock Parameter Scaling for Deep RL (ICML 2024 · similarity 0.85)
- Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Image Generation (NeurIPS 2026 · similarity 0.84)
- Representation Learning Enables Scalable Multitask Deep Reinforcement Learning (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.