Research map / Optimization / Implicit bias and generalization
Dimension-adapted Momentum Outscales SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a dimension- and time-adapted Nesterov-style momentum schedule (DANA) for stochastic momentum SGD on the power law random features model, showing it improves loss scaling exponents over SGD and SGD-M in the high-dimensional-line regime (2α>1), validated on PLRF simulations and 2-layer LSTMs on C4.
Problem
Whether hyperparameters of stochastic momentum algorithms can be chosen as functions of model size and data complexity to provably change the exponents of neural scaling laws, beyond SGD and fixed-momentum SGD-M.
Approach
Scale the momentum decay as δ(1+t)^{-1} and the momentum learning rate with dimension d and time (γ3 ∝ d^{-κ2}(1+t)^{-κ3}), where the time-dependent effective dimension (tB)^{-1/(2α)} replaces d in DANA-decaying; this changes the loss exponents rather than only constants, unlike fixed-momentum SGD-M which yields SGD's exponents.
Key results
- DANA-decaying (κ3=1/(2α)) outscales SGD and SGD-M in the 2α>1 regime for all training regimes 0<ℓ<2α, while SGD-M yields the same scaling exponents as SGD.
- On 2-layer LSTMs on C4, DANA-decaying loss exponents vary smoothly with κ3, outscaling SGD for 0.6 ≤ κ3 ≤ 0.9 and matching SGD at κ3 ≥ 1.0.
- DANA-decaying can be compute-optimal at a different d–t relationship than the Chinchilla d∝f^{1/2} law for 2α>1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dimension- and data-dependent DANA-decaying momentum instead of SGD | SGD on PLRF · PLRF, 2α > 1 and 0 < ℓ < 2α | loss exponent | Helped |
| Use DANA-decaying instead of DANA-constant momentum (ablation) | DANA on PLRF · PLRF, 2α > 1 and 0 < ℓ < 2α | loss exponent | Helped |
| Use fixed, non-scaling momentum (ablation) | SGD-M · PLRF, all regimes | loss exponent | No clear effect |
| Use DANA-constant momentum instead of SGD (ablation) | SGD on PLRF · PLRF, 2α > 1 and 1 < ℓ < 2α | loss exponent | Helped |
| Use DANA-constant momentum (ablation) | SGD on PLRF · PLRF, t ≤ d | loss exponent | No clear effect |
| Use DANA-constant momentum in Phase IIIb (ablation) | SGD on PLRF · PLRF, Phase IIIb and 2α > 1 | loss exponent | No clear effect |
| Set DANA-decaying κ₃ below 0.6 (ablation) | DANA-decaying on C4 · C4 language dataset | loss exponent | Hurt |
| Set DANA-decaying κ₃ in the range 0.6 to 0.9 (ablation) | DANA-decaying on C4 · C4 language dataset | loss exponent | Helped |
| Set DANA-decaying κ₃ ≥ 1.0 (ablation) | DANA-decaying on C4 · C4 language dataset | loss exponent | No clear effect |
| Set κ₃ = 1.0 (Schedule-Free SGD) (ablation) | DANA-decaying on C4 · C4 language dataset | loss exponent | No clear effect |
Surprising findings
- Fixed-momentum SGD-M produces identical scaling exponents to SGD; its speedup is only a constant-factor effective learning rate change.
- Sweeping the momentum learning-rate scale in DANA-constant to minimize loss yields the DANA-decaying schedule with κ3=1/(2α) on the PLRF.
Limitations
- Outscaling is only shown for 2α>1 (above the high-dimensional line); open whether it is possible for 2α<1.
- Theory is for quadratic/PLRF setting and deterministic equivalents; no convergence argument beyond quadratics for cross-entropy.
- Analysis does not cover adaptive methods such as Adam combined with DANA.
Experimental setup
Datasets: C4 synthetic power law random features data
Benchmarks: power law random features synthetic quadratics C4
Metrics: loss exponent parameter exponent data exponent compute-optimal loss
Baselines: SGD SGD-M DANA-constant Schedule-Free SGD AcSGD Adam with cosine decay
Builds on: SGD with momentum stochastic Nesterov acceleration Schedule-Free SGD accelerated SGD power law random features model
Closest papers by meaning
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.89)
- Gradient correlation is a key ingredient to accelerate SGD with momentum (ICLR 2025 · similarity 0.89)
- Random Scaling and Momentum for Non-smooth Non-convex Optimization (ICML 2024 · similarity 0.88)
- The Marginal Value of Momentum for Small Learning Rate SGD (ICLR 2024 · similarity 0.88)
- To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.87)
- Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.