Research map / Architectures and efficiency / RNNs and normalization
The Shattered Gradients Problem: If resnets are the answer, then what is the question?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use skip-connections with batch normalization and β-rescaling rather than a feedforward architecture | Deep rectifier networks · Typical datapoint pairs | gradient correlation: 1/2^L -> 1/(β√L) | Helped |
| Set β to 0.1 instead of 1.0 (ablation) | 100-layer resnet with batch normalization · Typical datapoint pairs | gradient correlation: 0.1 -> 0.7 | Helped |
| Use the looks-linear initialization with orthogonal weights | CReLU feedforward network · CIFAR-10 test | accuracy | Helped |
| Remove the looks-linear initialization (ablation) | CReLU feedforward network · CIFAR-10 test | accuracy | Hurt |
| Use β = 1.0 rather than β = 0.1 (ablation) | Resnet with batch normalization · 100-layer network | gradient correlation: 0.7 -> 0.1 | Hurt |
| Use β-rescaling in resnets (ablation) | Resnet with batch normalization · Typical datapoint pairs | gradient correlation: 1/(β√L) | Helped |
Closest papers by meaning
- How to Start Training: The Effect of Initialization and Architecture (NeurIPS 2018 · similarity 0.89)
- Towards Understanding the Importance of Shortcut Connections in Residual Networks (NeurIPS 2019 · similarity 0.88)
- Are ResNets Provably Better than Linear Predictors? (NeurIPS 2018 · similarity 0.88)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.87)
- On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning (NeurIPS 2025 · similarity 0.87)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.