Research map / Architectures and efficiency / RNNs and normalization
Super Consistency of Neural Network Landscapes and Learning Rate Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that under muP (and Depth-muP) the top Hessian eigenvalues (sharpness) are width-independent throughout training ("Super Consistency"), while NTK/standard/NTP scalings accumulate finite-size effects, and links this to feature learning and learning rate transfer.
Problem
Why learning rates transfer across width and depth under feature-learning parameterizations (muP / Depth-muP) when loss dynamics diverge across model sizes, and what landscape property underlies this optimization puzzle.
Approach
Introduces "Super Consistency": the top Hessian eigenvalues (sharpness) under muP converge to a width-independent threshold and do not accumulate finite-size deviations over training, unlike loss and NTK eigenvalues; the sharpness dynamics are attributed to feature learning via a Gauss-Newton/NTK decomposition and analyzed on a two-layer linear network.
Key results
- Under muP, top Hessian eigenvalues converge to a width-independent threshold and remain consistent over training, while training loss and NTK eigenvalues diverge with width (power-law fit beta > 0.6 for loss, beta approx 2 for NTK top eigenvalue).
- Under NTP, the sharpness decreases with width and sharpness dynamics separate, correlating with absence of learning rate transfer.
- Depth-muP with one layer per residual block shows sharpness super consistency across depth and learning rate transfer; multi-layer blocks (e.g. ViT, two-layer ConvNet blocks) show finite-size accumulation and no transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use μP instead of NTK parameterization | ConvNet · CIFAR-10 | sharpness consistency across widths during training | Helped |
| use μP or Depth-μP instead of parameterizations without learning-rate transfer | neural networks across widths and depths · CIFAR-10 and other reported settings | learning-rate transfer | Helped |
| use Depth-μP scaling across depth | residual networks with one layer per block · CIFAR-10 | sharpness consistency across depth | Helped |
| use multiple layers per residual block instead of one (ablation) | Depth-μP residual networks · CIFAR-10 | sharpness consistency during training | Hurt |
| use multiple layers per residual block instead of one (ablation) | Depth-μP residual networks · CIFAR-10 | learning-rate transfer | Hurt |
| use standard parameterization instead of a feature-learning scaling | ConvNets · CIFAR-10 | learning-rate transfer | Hurt |
| add linear learning-rate warmup (ablation) | μP networks without 1/√depth residual scaling · CIFAR-10 | trainability at larger learning rates | Helped |
| add linear learning-rate warmup (ablation) | μP networks without 1/√depth residual scaling · CIFAR-10 | learning-rate transfer | Mixed |
| disable residual connections (ablation) | μP ConvNets · CIFAR-10 | trainability as depth increases | Hurt |
| disable residual connections (ablation) | μP ConvNets · CIFAR-10 | sharpness consistency across depth | No clear effect |
| increase batch size (ablation) | μP networks · reported batch-size experiments | learning-rate transfer | No clear effect |
| train with full-batch gradient descent (ablation) | μP residual ConvNets · CIFAR-10 5000-sample subset | learning-rate transfer | Helped |
Surprising findings
- Loss curves diverge across widths under muP even though learning rate transfers, indicating the sharpness rather than the loss is the consistent quantity.
- Under muP the sharpness trajectory is width-independent even though NTK eigenvalues accumulate finite-size effects over time.
Limitations
- Hessian estimation is computationally expensive, so experiments are not run at larger scale than presented.
- Theory on Super Consistency does not exclude other factors affecting optimal learning rate; sharpness-learning rate relationship is assumed.
- Theoretical analysis is restricted to a two-layer linear network with simplified data and MSE loss.
Experimental setup
Datasets: CIFAR-10 Tiny-ImageNet ImageNet WikiText
Benchmarks: CIFAR-10 Tiny-ImageNet WikiText
Metrics: sharpness (top Hessian eigenvalue) training loss NTK eigenvalues learning rate transfer
Baselines: Neural Tangent Parametrization Standard Parametrization Depth-muP with multiple layers per block
Builds on: muP (Tensor Programs) Depth-muP Neural Tangent Kernel Edge of Stability Gauss-Newton decomposition
Closest papers by meaning
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.89)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.88)
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.88)
- The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.88)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.88)
- Simplicity Bias via Global Convergence of Sharpness Minimization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.