Research map / Optimization / Optimizers
Scaling Exponents Across Parameterizations and Optimizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies width-scaling parameterizations (standard, NTK, muP, mean-field) with SGD, Adam and Adafactor-style parameter scaling, measuring alignment empirically and proposing per-layer learning rate and epsilon prescriptions, plus Adam-atan2.
Problem
Prior width-scaling theory for learning rates relies on a strong assumption that parameter updates are aligned with the data, which may be too conservative and exclude the best-performing learning rate scalings; epsilon in Adam also breaks scale invariance at large width.
Approach
Generalizes the stability/nontriviality constraints with explicit alignment exponents (alpha, omega, u), derives max stable per-layer learning rates under "full alignment" and "no alignment" assumptions for all parameterizations and optimizers, measures a log alignment ratio during training, and adds per-layer constant multipliers tuned at small scale and transferred to large scale.
Key results
- Best 26.8B model for Adam is standard parameterization with per-layer full-alignment learning rates and optimal constants (embedding LR O(1), hidden and readout O(1/n)), with eval loss 2.572, outperforming muP's best 2.599.
- The 15.3B standard parameterization model outperforms the 26.8B muP model under the per-layer prescription.
- Adam with parameter scaling and NTK under no-alignment exponents gives the best overall eval loss of 2.566 at 26.8B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full-alignment per-layer learning rates instead of global learning rates (ablation) | Adam + standard parameterization · C4 eval, 26.8B parameters | eval loss: 2.625 -> 2.576 | Helped |
| Use full-alignment per-layer learning rates instead of global learning rates (ablation) | Adam + muP · C4 eval, 26.8B parameters | eval loss: 2.727 -> 2.646 | Helped |
| Use full-alignment per-layer learning rates instead of global learning rates (ablation) | Adam + MFP · C4 eval, 26.8B parameters | eval loss: 2.638 -> 2.630 | Hurt |
| Tune per-layer constant learning-rate multipliers (ablation) | Adam per-layer learning rates · C4 eval, across parameterizations and model sizes | eval loss | Helped |
| Use no-alignment instead of full-alignment per-layer learning-rate exponents (ablation) | Adam + NTK, muP, and MFP · C4 eval, across model sizes | eval loss | Helped |
| Use full-alignment per-layer learning-rate exponents instead of global learning rates (ablation) | SGD · C4 eval, across parameterizations | eval loss | No clear effect |
| Tune per-layer constant learning-rate multipliers (ablation) | SGD per-layer learning rates · C4 eval, across parameterizations | eval loss | Helped |
| Add parameter scaling to Adam (ablation) | Adam across parameterizations · C4 eval, best setting per parameterization | eval loss | Mixed |
| Use no-alignment (global) instead of full-alignment per-layer learning-rate exponents (ablation) | Adam + parameter scaling · C4 eval, across parameterizations and model sizes | eval loss | Helped |
| Use optimal rather than default per-layer constants (ablation) | Adam + parameter scaling with full-alignment exponents · C4 eval, across model sizes | eval loss | Hurt |
| Use Adam-atan2 to eliminate epsilon | Adaptive optimizers · C4 eval | eval loss | Helped |
Surprising findings
- Standard parameterization with per-layer learning rates outperforms muP for Adam, contrary to prior comparisons.
- Adam with parameter scaling makes the no-alignment exponents equivalent to global learning rate, and all parameterizations behave similarly.
Limitations
- Theory assumes infinite precision and O(1) depth, training steps and batch size one, and the alignment assumptions are only two extremes.
- Experiments use fixed 50,000 steps without weight decay; compute-optimal horizons may change exponents.
- SGD results are noisy and hard to interpret.
Experimental setup
Datasets: C4
Benchmarks: C4 eval loss
Metrics: eval loss log alignment ratio power-law exponent of optimal learning rate
Baselines: standard parameterization NTK parameterization muP mean-field parameterization global learning rate Adam
Builds on: muP Tensor Programs Adafactor NanoDO
Closest papers by meaning
- On the Provable Separation of Scales in Maximal Update Parameterization (ICML 2025 · similarity 0.89)
- How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization (NeurIPS 2026 · similarity 0.89)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
- Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling (NeurIPS 2024 · similarity 0.88)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.