Research map / Optimization / Implicit bias and generalization
Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width
Dayal Singh Kalra, Maissam Barkeshli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase depth d and 1/width | SGD on FCN/CNN/ResNet, MSE · early training phase diagram | c_loss, c_sharp: c_loss 2.82 -> 36.75 | Improved |
| Increase depth and 1/width opens a sharpness-reduction phase between c_crit and c_loss | SGD on DNNs · early training | sharpness | Improved |
| Center the network output at initialization (f - f(theta0)) (ablation) | FCN d=8 CIFAR-10 MSE · phase diagram | sharpness reduction phase / c_loss | Improved |
| Set last layer weights to zero at initialization (ablation) | FCN d=8 CIFAR-10 MSE · phase diagram | sharpness reduction phase / c_loss | Improved |
| Scale the output by a fixed factor alpha (ablation) | FCN · early training | c_loss growth with d and 1/w | No clear effect |
| Use cross-entropy instead of MSE | SGD · early training | c_loss and c_crit: c_crit 2 -> 4 >= c_crit >= 2 | Mixed |
| Reduce batch size from 512 to 32 (ablation) | SGD · early training phase diagram | phase diagram | No clear effect |
| Add bias (ablation) | SGD networks · early training | phase diagram | No clear effect |
| Increase width of uv model from w=2 to w=512 (ablation) | uv model · MSE, y=0 | c_loss: 3.74 -> 2.1 | Improved |
| Increase learning rate constant c above c_crit | 7-layer CNN · CIFAR-10 sharpness at c tau=200 | normalized sharpness | Improved |
| Increase learning rate between c_loss and c_barrier | CNNs · linear interpolation of loss | loss barrier | No clear effect |
Closest papers by meaning
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.91)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.90)
- Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More (ICML 2025 · similarity 0.89)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.88)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.88)
- 4+3 Phases of Compute-Optimal Neural Scaling Laws (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
