Research map / Architectures and efficiency / RNNs and normalization
On the Provable Separation of Scales in Maximal Update Parameterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of Maximal Update Parameterization (µP) showing loss landscapes (macro-variables) converge at O(1/n) with width while weights (micro-variables) evolve at Θ(1/√n), explaining early-stage hyperparameter tuning and learning-rate delay effects.
Problem
Theoretical understanding of why hyperparameters tuned in early training under µP approximately transfer across model widths and to full training remains lacking.
Approach
Uses an SDE framework under µP with a width-dominance regime to separate macro-variables (loss, which converges at O(1/n)) from micro-variables (weights, Θ(1/√n)), yielding an O(1/n) bound on hyperparameter transfer error and a scaling-law account of learning-rate delay.
Key results
- Proves macro-variable (loss) width-convergence rate O(1/n) under µP (Proposition 4.6, Corollary 4.7).
- Shows micro-variable weight displacement satisfies E||µ(t)|| = Θ(1/√n) (Theorem 4.1, Lemma 4.10).
- Derives transfer error bound O(1/min(n1,n2)) for optimal learning rate across widths (Proposition 4.8, Lemma 4.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use µP with width-scaled learning rates | Neural networks across widths · µP neural networks, width n | loss landscape convergence rate: O(1/n) | Helped |
| Use µP with width-scaled learning rates | Neural network weights · µP neural networks, fixed t ∈ [0, T] | expected weight displacement: Θ(1/√n) | Mixed |
| Transfer the optimal learning rate between widths | µP neural networks · Widths n1 and n2 | learning-rate transfer error: O(1/ min(n1, n2)) | Helped |
| Tune hyperparameters during early-stage training | µP neural networks · Early-stage training versus full training | hyperparameter transfer error: O(1/n) | Helped |
| Include an integrated learning-rate memory descriptor | Infinite-width µP neural networks · Training loss under time-varying learning rate | training loss: L(t) = L0 + AS1(t)−α − CS2(t) | Mixed |
Limitations
- Relies on width-dominance (n >> T) and strong assumptions such as L-smoothness and width regularity.
- Scope limited to µP; does not cover adaptive optimizers like Adam or joint width-depth scaling.
- Empirical validation is not provided in the paper body.
Experimental setup
Builds on: Maximal Update Parameterization (µP) Xie et al. 2024 SDE framework for hyperparameter scaling neural tangent kernel mean-field theory
Closest papers by meaning
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.89)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.89)
- u-$\mu$P: The Unit-Scaled Maximal Update Parametrization (ICLR 2025 · similarity 0.89)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.88)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.87)
- Super Consistency of Neural Network Landscapes and Learning Rate Transfer (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.