Research map / Optimization / Optimizers
The Road Less Scheduled
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Schedule-Free optimization, which replaces learning-rate schedules with an interpolation of gradient-evaluation and iterate-averaging sequences, matching or beating tuned schedules across 28 problems and winning the AlgoPerf self-tuning track.
Problem
Learning rate schedules require fixing the optimization stopping time T in advance, while schedule-free alternatives (Polyak and primal averaging) perform much worse in practice despite theoretical optimality.
Approach
Interpolate between the base SGD/Adam iterate z and the averaged iterate x (y = (1-β)z + βx, gradients at y, x as online equal-weighted average of z), with a new online-to-batch conversion theorem that unifies prior averaging schemes and implicitly applies a linear-decay schedule without knowing T.
Key results
- Schedule-Free SGD and AdamW match or exceed tuned cosine schedules on CIFAR-10, CIFAR-100, SVHN, IWSLT-14 and OpenWebText GPT-2, and tie on ImageNet and MAE fine-tuning.
- Schedule-Free AdamW won the MLCommons 2024 AlgoPerf Self-Tuning track, outperforming NAdamW on WMT, VIT, FASTMRI and OGBG.
- On 12 LibSVM logistic regression problems, Schedule-Free converges faster on all but one and has higher accuracy on 6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace cosine scheduling with Schedule-Free SGD | CIFAR10, CIFAR100, and SVHN · CIFAR10, CIFAR100, and SVHN | test performance | Helped |
| Replace cosine scheduling with Schedule-Free AdamW | IWSLT14 and OpenWebText GPT-2 · IWSLT14 and OpenWebText | evaluation performance | Helped |
| Replace Linear Decay scheduling with Schedule-Free AdamW | fastMRI and Criteo DLRM · fastMRI and Criteo DLRM | evaluation performance | Helped |
| Replace the best schedule with Schedule-Free training | MAE fine-tuning and ImageNet ResNet-50 · MAE fine-tuning and ImageNet | test performance | No clear effect |
| Replace NAdamW with Schedule-Free AdamW | AlgoPerf WMT, VIT, FASTMRI, OGBG, Conformer, Criteo, and DeepSpeech · AlgoPerf self-tuning track | time-to-target | Mixed |
| Replace averaging approaches and Linear Decay with Schedule-Free learning | LibSVM logistic regression problems · LibSVM logistic regression | final train accuracy | Helped |
| Use momentum β=0.9 instead of β=0.98 (ablation) | Schedule-Free AdamW · NanoGPT on OpenWebText | loss | Hurt |
| Use β<1 rather than β=1 (ablation) | Schedule-Free momentum · Quadratic minimization | convergence | Helped |
| Compute weight decay at y rather than z (ablation) | Schedule-Free methods · ImageNet and NanoGPT | performance | Helped |
| Use learning-rate-squared weighted averaging during warmup instead of unweighted averaging (ablation) | Schedule-Free AdamW · Deep learning tasks with warmup | performance | Helped |
Surprising findings
- Exponential-moving-average momentum can hurt worst-case convergence in the non-smooth setting, while the interpolation form does not, for any beta in [0,1].
- Large fixed step sizes D/G give better practical performance than the theory-suggested D/(G sqrt(t)), and the paper proves optimal rates hold in a special case under a regret condition.
Limitations
- Learning rate and weight decay still need sweeping, and the best values differ from those used with a schedule.
- Batch-norm models require extra handling (recomputing running statistics on the x sequence), which the AlgoPerf setup did not easily allow.
- Theoretical understanding of when large learning rates work for stochastic problems is incomplete.
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN ImageNet IWSLT14 Criteo Kaggle fastMRI OpenWebText WMT17 LibriSpeech Criteo 1B OGBG PCBA
Benchmarks: CIFAR10 CIFAR100 SVHN ImageNet IWSLT14 Criteo Kaggle fastMRI OpenWebText NanoGPT MLCommons AlgoPerf self-tuning LibSVM logistic regression
Metrics: test accuracy validation loss time-to-target train accuracy
Baselines: cosine learning rate schedule linear decay schedule step-wise schedule NAdamW Polyak averaging primal averaging
Builds on: Polyak-Ruppert averaging primal averaging AdamW Lookahead Nesterov accelerated method
Closest papers by meaning
- Taking the Road Less Scheduled with Adaptive Polyak Steps (NeurIPS 2026 · similarity 0.89)
- The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training (ICML 2025 · similarity 0.88)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.85)
- Accelerating neural network training: An analysis of the AlgoPerf competition (ICLR 2025 · similarity 0.85)
- Stepsize anything: A unified learning rate schedule for budgeted-iteration training (NeurIPS 2025 · similarity 0.85)
- Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.