Research map / Reinforcement learning / Policy gradient and actor-critic
SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimBa, a network architecture (running-statistics observation normalization, pre-LN residual feedforward blocks, post-LayerNorm) that injects simplicity bias so deep RL agents can scale parameters, improving SAC, DDPG, TD-MPC2, PPO and METRA.
Problem
Scaling network parameters in deep RL often degrades performance (overfitting / loss of plasticity), unlike CV and NLP where scaling succeeds; the paper asks how architecture can enable effective parameter scaling in RL.
Approach
Embed simplicity bias architecturally via running-statistics observation normalization (RSNorm), a residual feedforward block with a direct linear input-to-output pathway and pre-LayerNorm, and a post-LayerNorm before the output head; scale width of the critic. Simplicity is measured by Fourier-based complexity of randomly initialized networks.
Key results
- SAC with SimBa improves average return on DMC-Hard by 570 points over the MLP version; DDPG by 480 and TD-MPC2 by 170.
- SimBa (SAC + SimBa) matches or surpasses state-of-the-art off-policy methods across 51 continuous control tasks while requiring less compute than BRO and TD-MPC2 in most settings.
- Across 12 architectures on DMC-Hard, simplicity bias score correlates with return with Pearson rho=0.79 for RSNorm-based architectures and 0.54 across all.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MLP actor-critic networks with SimBa | SAC · DMC-Hard | average return: increased the average return by 570 points | Helped |
| Replace MLP actor-critic networks with SimBa | DDPG · DMC-Hard | average return: increased the average return by 480 points | Helped |
| Replace the shared MLP encoder with SimBa | TD-MPC2 · DMC-Hard | average return: increased the average return by 170 points | Helped |
| Add residual connections | SAC with MLP · DMC-Hard, 1M steps | average return: increases the average return by 50 points | Helped |
| Add layer normalization | SAC with MLP · DMC-Hard, 1M steps | average return: adds 150 points | Helped |
| Add all SimBa components | SAC with MLP · DMC-Hard, 1M steps | average return: improvement of 550 points | Helped |
| Remove residual connections, pre-layer normalization, or post-layer normalization (ablation) | SAC + SimBa · DMC-Hard | average return | Hurt |
| Remove RSNorm (ablation) | SAC + SimBa · DMC-Hard | average return | Hurt |
| Replace RSNorm with LayerNorm or BatchNorm (ablation) | SAC + SimBa · DMC-Hard | performance | No clear effect |
| Replace RSNorm with Env Wrapper RSNorm (ablation) | SAC + SimBa · DMC-Hard | performance | Hurt |
| Use fixed initial observation statistics instead of adaptive RSNorm (ablation) | SAC + SimBa · DMC-Hard | performance | Hurt |
| Scale actor width or depth (ablation) | SAC + SimBa · DMC-Hard | performance | Hurt |
Surprising findings
- Removing RSNorm reduces performance strongly yet increases the simplicity bias score, suggesting the uniform-input Fourier measure does not capture RSNorm's contribution.
- Scaling actor width or depth tends to reduce performance while scaling the critic helps, suggesting the actor needs lower complexity.
Limitations
- Simplicity bias measure based on uniform-input Fourier analysis does not capture RSNorm's contribution.
- Correlation between simplicity bias and performance is weaker across all architectures (rho=0.54) than for RSNorm-based ones.
- Study is limited to state-based inputs; vision-based RL and multi-task settings are left for future work.
Experimental setup
Datasets: DeepMind Control Suite MyoSuite HumanoidBench Craftax
Benchmarks: DeepMind Control Suite MyoSuite HumanoidBench Craftax
Metrics: average episode return average success rate average score simplicity bias score state coverage sample efficiency compute time
Baselines: SAC DDPG TD7 BRO TD-MPC2 DreamerV3 PPO METRA BroNet SpectralNet
Builds on: Soft Actor-Critic BRO TD-MPC2 PPO METRA Neural Redshift
Closest papers by meaning
- Hyperspherical Normalization for Scalable Deep Reinforcement Learning (ICML 2025 · similarity 0.89)
- Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning (ICML 2025 · similarity 0.86)
- Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.86)
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.86)
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.