Research map / Architectures and efficiency / RNNs and normalization
Mean Field Residual Networks: On the Edge of Chaos
Ge Yang, Samuel Schoenholz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase Tanh network depth while keeping initialization variance fixed | Tanh residual networks · MNIST | test accuracy | Hurt |
| Scale Tanh initialization variance with depth, following the predicted gradient-explosion level curve | Tanh residual networks · MNIST | test accuracy | Improved |
| Increase the skip-bias to residual-weight variance ratio while holding the leading gradient-growth term fixed | Tanh/FRN · MNIST | test accuracy | Mixed |
| Increase ReLU network depth toward the onset of numerical overflow | ReLU/FRN · MNIST | test accuracy | Improved |
| Increase α and depth to maximize metric expressivity without numerical failure | α-ReLU/FRN · MNIST | test accuracy | Improved |
| Use gradient explosion as the initialization-selection criterion for ReLU or α-ReLU | ReLU and α-ReLU residual networks · MNIST | test accuracy | No clear effect |
| Use Xavier or He initialization without adapting variances to depth | Residual networks · Initialization quality | test performance | Hurt |
| Use the mean-field theory to predict behavior from initialization hyperparameters | Residual networks · MNIST | relative test performance prediction | Improved |
Closest papers by meaning
- A Mean Field Analysis Of Deep ResNet And Beyond: Towards Provably Optimization Via Overparameterization From Depth (ICML 2020 · similarity 0.90)
- Doubly infinite residual neural networks: a diffusion process approach (JMLR 2021 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- Implicit regularization of deep residual networks towards neural ODEs (ICLR 2024 · similarity 0.87)
- Uniform-in-Time Weak Propagation of Chaos in Shallow Neural Networks (NeurIPS 2026 · similarity 0.87)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
