Research map / Architectures and efficiency / RNNs and normalization
How to Start Training: The Effect of Initialization and Architecture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize ReLU weights from a symmetric distribution with variance 2/fan-in | Fully connected ReLU networks · MNIST | epochs to 20% test accuracy | Helped |
| Use weight variance below 2/fan-in | Fully connected ReLU networks · MNIST | epochs to 20% test accuracy | Hurt |
| Use weight variance above 2/fan-in | Fully connected ReLU networks · MNIST | epochs to 20% test accuracy | Hurt |
| Use truncated He normal instead of an initialization with variance 2/fan-in | Fully connected ReLU networks · MNIST | mean square activation length | Hurt |
| Scale residual modules so that the sum of their scales is finite | ResNets · MNIST | epochs to 20% test accuracy | Helped |
| Keep each residual-module scale at 1 as depth increases | ResNets · MNIST | mean squared activation length | Hurt |
| Increase fully connected layer widths to reduce the sum of reciprocal widths | Fully connected ReLU networks · MNIST | epochs to 20% test accuracy | Helped |
| Permute layer widths while preserving their sum of reciprocals | Fully connected ReLU networks · MNIST | early training dynamics | No clear effect |
| Use constant width as depth increases, increasing the sum of reciprocal widths | Fully connected ReLU networks · MNIST | epochs to 20% test accuracy | Hurt |
| Initialize convolutional weights with variance 2/fan-in | Convolutional ReLU networks · CIFAR-10 | mean output length | Helped |
| Avoid failure mode FM1 by correctly scaling residual modules | ResNets · Residual-network activation lengths | empirical variance of activation lengths | Helped |
Closest papers by meaning
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- Initial Guessing Bias: How Untrained Networks Favor Some Classes (ICML 2024 · similarity 0.87)
- If You Want to Be Robust, Be Wary of Initialization (NeurIPS 2024 · similarity 0.87)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.87)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.87)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.