Research map / Optimization / Implicit bias and generalization
On the Weight Dynamics of Deep Normalized Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models the early-training dynamics of weight and gradient norms in normalized networks via ODEs, predicting that effective learning rate ratios across layers equalize in gradient flow but can flip or fail to converge at large step sizes, and validates this with ResNet/Transformer experiments and an ELR-constraining scheduler.
Problem
Layer-wise effective learning rates (gradient-to-weight norm ratios) in deep normalized networks can differ by orders of magnitude at initialization, causing instability and frozen layers, and the time evolution of these disparities is not well understood.
Approach
Assume weights and gradients are random with time-dependent variance, use scale invariance (orthogonality of gradient and weights) to derive a discrete recurrence and its continuous ODE limit for weight norm growth, predicting ELR equalization in gradient flow and discrete overshooting/flipping at larger learning rates; extend to momentum and warm-up, and use a gradient rescaling scheme that fixes
Key results
- In the continuous gradient flow, the squared ELR ratio between layers tends to 1 as t grows, so ELRs equalize despite exponential initial disparity.
- Using the ELR-constrained training on a ResNet110 without skip connections on CIFAR-10, the best accuracy reached is 73.9% versus 34.0% for the best conventional run.
- Warm-up consistently lowers the average weighted ELR spread for ResNet110 Short on CINIC-10 and yields faster training and higher final accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Constrain layerwise effective learning rates | ResNet110 NoShort · CIFAR-10 test | accuracy: 34.0% -> 73.9% | Helped |
| Constrain layerwise effective learning rates | ResNet110 Short · CIFAR-100 test | accuracy: 66.5% -> 71.7%; 73.6% | Mixed |
| Constrain layerwise effective learning rates | ResNet110 NoShort · CIFAR-100 test | accuracy: 7.3% -> 28.1% | Helped |
| Add a learning-rate warm-up phase (ablation) | ResNet110 Short · CINIC-10, 50 epochs | validation accuracy | Helped |
| Increase momentum (ablation) | ResNet56 Short · CIFAR-10 | training accuracy | Helped |
| Increase momentum (ablation) | ResNet56 Short · CIFAR-10 | relative ELR spread | Helped |
| Add a warm-up phase (ablation) | ResNet110 Short · CINIC-10, 50 epochs | relative ELR spread | Helped |
| Use constrained ELR training instead of regular training | Transformer · Multi30k translation | perplexity | Helped |
| Remove the warm-up phase (ablation) | Transformer with SGD, momentum, and weight decay · Multi30k translation | perplexity | Hurt |
| Use constrained ELR training without warm-up (ablation) | Transformer · Multi30k translation | perplexity | Helped |
Surprising findings
- For large learning rates, the lowest layers' ELRs flip below higher layers after a single step, so ratios do not converge to 1 at constant step size in the discrete model.
- Without residual connections, the lower layers in deep networks can saturate and never recover, so high ELR spread predicts low trainability.
Limitations
- Model assumes constant base gradient magnitude during early training, which is violated after a few steps (e.g. ToyNet50 ELRs drop lower than predicted).
- Momentum extension relies on strong assumptions such as i.i.d. random base gradients.
- Long-term predictions are less accurate when real gradients and affine BatchNorm parameters are present.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CINIC-10 ImageNet (ILSVRC 2012) Multi30k
Benchmarks: CIFAR-10 CIFAR-100 CINIC-10 Multi30k
Metrics: test accuracy validation accuracy perplexity relative ELR spread
Baselines: regular SGD training SGD with momentum and weight decay OneCycle warm-up Adam (Transformer baseline)
Builds on: batch normalization weight normalization LARS He initialization OneCycle learning rate schedule
Closest papers by meaning
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.88)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.87)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.87)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.