Research map / Reinforcement learning / Policy gradient and actor-critic

Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Alternative policy update rules · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that gradient vanishing under non-stationarity causes scaling failures in deep RL, and fixes it with multi-skip residual MLPs plus the Kron (Kronecker-factored) optimizer, improving PQN, PPO, Simba and offline Q-learning at scale.

Problem

Deep RL networks degrade in performance when scaled in depth and width; the root causes, especially the interplay of non-stationarity and gradient pathologies, are poorly understood.

Approach

Diagnose scaling failure as gradient degradation under non-stationarity, then stabilize gradient flow with two simple interventions: multi-skip residual connections broadcasting encoder features to all MLP blocks, and a second-order Kronecker-factored optimizer (Kron) replacing RAdam/Adam.

Key results

• Combined interventions on PQN outperform the baseline in 90% of 57 ALE games, with a median relative improvement of 83.27%.
• Applying the same interventions to PPO yields a median improvement of 31.40% on the full ALE and outperforms baseline in 83.64% of games.
• Kron optimizer prevents performance collapse of deep MLPs in PQN on Atari-10, and multi-skip residuals keep PQN trainable across all scales studied.

Measured effects reported in the paper

Surprising findings

• The baseline PQN is competitive with strong agents such as Rainbow, even before the interventions are applied.
• Several widely used alternatives (cosine/cyclic LR, GELU, CReLU, L2 norm, weight clipping, weight decay) did not consistently improve scaling performance compared with the proposed combination.

Limitations

• Computational resources limited exploration of architectures beyond a certain size.
• Second-order Kron optimizer adds significant computational overhead relative to Adam/RAdam.
• Proposed interventions are described by the authors as not necessarily optimal, serving as a baseline.

Experimental setup

Datasets: Atari-10 ALE (57 games) CIFAR-10 IsaacGym Cartpole and Anymal DeepMind Control Suite (Humanoid Walk/Run, Dog Trot/Run) OGBench offline RL tasks

Benchmarks: Atari-10 ALE full suite (57 games) IsaacGym Cartpole and Anymal DMC humanoid and dog tasks CIFAR-10 non-stationary label shuffling offline Q-learning tasks (humanoidmaze-giant-navigate, puzzle-4x5-play, puzzle-4x6-play)

Metrics: human-normalized score median human-normalized score episode return gradient norm dormant neuron fraction SRank Hessian trace

Baselines: PQN with RAdam PPO DQN Rainbow Simba with AdamW DDPG SAC SHARSA DSHARSA

Builds on: PQN Simba K-FAC Rainbow PPO

Closest papers by meaning

• Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning (ICML 2025 · similarity 0.89)
• Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Understanding and Diagnosing Deep Reinforcement Learning (ICML 2024 · similarity 0.87)
• Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control (NeurIPS 2024 · similarity 0.86)
• Stabilizing Policy Optimization via Logits Convexity (NeurIPS 2026 · similarity 0.86)
• Target-Aligned Reinforcement Learning (NeurIPS 2026 · similarity 0.86)