Research map / Architectures and efficiency / RNNs and normalization
Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?
Boris Hanin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the sum of reciprocals of hidden-layer widths, β | Random fully connected ReLU networks at initialization · Fully connected ReLU networks | Variance of squared Jacobian entries: exponential in \beta | Hurt |
| Increase hidden-layer widths so that \sum_{j=1}^{\infty}\frac{1}{n_j}<\infty | Random fully connected ReLU networks at initialization · Deep fully connected ReLU networks | Gradient stability: \sum_{j=1}^{\infty}\frac{1}{n_j}<\infty | Improved |
| Make hidden-layer widths equal while holding depth and total hidden-neuron budget fixed | Fully connected ReLU networks · Deep fully connected ReLU networks | Gradient stability | Improved |
| Vary the weight distributions while keeping weights independent, zero-centered, and correctly variance-scaled | Random fully connected ReLU networks · Fully connected ReLU networks | Occurrence of the EVGP | No clear effect |
Closest papers by meaning
- Convergence of the Gradient Flow for Shallow ReLU Networks on Weakly Interacting Data (NeurIPS 2025 · similarity 0.88)
- Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies (JMLR 2024 · similarity 0.88)
- On the spectral bias of two-layer linear networks (NeurIPS 2023 · similarity 0.88)
- Mean Field Residual Networks: On the Edge of Chaos (NeurIPS 2017 · similarity 0.88)
- Uniform-in-Time Weak Propagation of Chaos in Shallow Neural Networks (NeurIPS 2026 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
