Research map / Optimization / Optimizers

Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers

NeurIPS 2026 · Optimization › Optimizers · approach: Second-order and structured preconditioning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a suboptimality-dependent generalized smoothness assumption for LMO-based optimizers (Muon, Lion, normSGD) that yields a theoretical warm-up-then-decay learning rate schedule and a practical adaptive warm-up scheduler that sets its own warm-up length.

Problem

Learning rate warm-up is used ubiquitously for norm-constrained (LMO-based) optimizers but lacks theoretical justification, and its duration is typically hand-tuned.

Approach

Assume local smoothness of the LMO gradient-change ratio is K(x)=K0+K1*Delta+K_rho*Delta^rho, which gives a step size eta=Delta/(D*K(x)) that naturally warms up and then decays; a practical scheduler fits a rational schedule to a warmup+decay target via three constraints and a Gaussian-weighted matching over Delta'.

Key results

• On LLaMA 124M with Muon (bs=32), the adaptive scheduler matches or beats the best manually tuned warm-up length without grid search.
• On Swin-B ImageNet-1K with Muon, the adaptive scheduler surpasses the best manually tuned top-1 accuracy.
• Ablations show any target loss f* in [3.0, 4.0] and sigma_F^2 in [1e-3, 3e-3] outperform the best manual warm-up on LLaMA 124M.

Surprising findings

• Very small Frobenius variance (sigma_F^2 <= 1e-4) degrades performance, and the method is sensitive to extreme target loss values (f*=0 or >=5).

Limitations

• Requires a prior estimate of the target loss f*, which the paper says is the only parameter needing task knowledge.
• Theory relies on global assumptions (star-convexity, bounded iterates); stochastic results need interpolation and bounded suboptimality M.
• Experiments are limited to small LLaMA models (124M/210M) and one vision setting; the empirical Lion/Muon results rely on a few hyperparameter choices.

Experimental setup

Datasets: FineWeb ImageNet-1K

Benchmarks: FineWeb validation loss ImageNet-1K top-1 accuracy

Metrics: validation loss top-1 accuracy

Baselines: manually tuned linear warm-up + cosine decay WSD WSO

Builds on: Muon Lion (L0,L1)-smoothness Edge of Stability Schedule-Free WSD schedule

Closest papers by meaning

• MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.87)
• The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training (ICML 2025 · similarity 0.87)
• Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training (NeurIPS 2024 · similarity 0.87)
• Why Warmup the Learning Rate? Underlying Mechanisms and Improvements (NeurIPS 2024 · similarity 0.85)
• When Will Gradient Regularization Be Harmful? (ICML 2024 · similarity 0.84)
• The Road Less Scheduled (NeurIPS 2024 · similarity 0.84)