Research map / Architectures and efficiency / RNNs and normalization
Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows in a large-scale study that linear decay-to-zero (D2Z) LR schedules outperform the standard 10x cosine decay for LLM pretraining at compute-optimal and over-trained token budgets, and explains this via an EMA view of AdamW.
Problem
Which learning rate schedule minimizes loss for LLM pretraining at a fixed number of training tokens, given that the standard practice is warmup followed by cosine decay to 10% of peak LR.
Approach
Interprets AdamW weights as an exponential moving average of weight updates with time-varying smoothing set by the LR schedule, showing linear decay to zero balances bias reduction early and gradient-noise averaging late; empirically finds linear D2Z optimal across peak LRs, model sizes, batch sizes, and TPP.
Key results
- A 610M model trained for 80 TPP with linear D2Z achieves lower validation loss than the same model trained for 200 TPP with 10x decay, corresponding to about 60% compute savings.
- At 20 TPP, optimal linear D2Z achieves 0.77% lower loss than optimal linear 10x decay for 610M models.
- At 1.7B scale trained to 80 TPP, D2Z achieves roughly 3.05% lower loss than 10x decay at the proxy-tuned peak LR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use linear decay to zero instead of linear decay to 10% of peak | 610M GPT-like model · SlimPajama, 20 TPP | validation loss: 0.77% lower loss | Helped |
| Use linear rather than cosine decay to zero | 610M GPT-like model · SlimPajama, 20 TPP | validation loss: 2.574 -> 2.571 | Helped |
| Decay to zero instead of 10% of peak | 111M and 610M models · 2 TPP | validation loss | Hurt |
| Decay to zero instead of 10% of peak | 111M models · 200 TPP | validation loss: 1.6% | Helped |
| Decay to zero instead of 10% of peak | 610M model · 200 TPP | validation loss: 2.6% | Helped |
| Train with D2Z rather than 10x decay | 1.7B model · SlimPajama, 80 TPP | loss: roughly 3.05% lower loss | Helped |
| Use D2Z instead of WSD | GPT-like model · 80 TPP | loss: around 0.84% better | Helped |
| Use D2Z instead of 10x decay | 610M GPT-like model · Downstream average, 80 TPP | Downstream average: 43.05% -> 43.54% | Helped |
| Use D2Z instead of 10x decay | 610M GPT-like model · Downstream average, 200 TPP | Downstream average: 43.46% -> 45.00% | Helped |
| Add step decay to a constant learning-rate schedule (ablation) | 111M model · SlimPajama, 20 TPP | validation loss | Helped |
| Add step decay to a 10x or D2Z schedule (ablation) | 111M model · SlimPajama, 20 TPP | validation loss | Hurt |
| Increase weight decay (ablation) | D2Z schedule · SlimPajama, 610M and 111M, 20 TPP | validation loss | Helped |
Surprising findings
- Cosine 10x decay is consistently slightly better than linear 10x decay, so linear alone is not the key; linear plus decay-to-zero is.
- Constant and 10x schedules overfit to the final portion of training data, while D2Z does not.
Limitations
- Requires knowing total training steps in advance.
- Focused on AdamW and LLM pretraining at compute-optimal/over-trained sizes; less applicable with limited data.
- May not help unstable training that cannot tolerate high learning rates.
Experimental setup
Datasets: SlimPajama OpenWebText
Benchmarks: MMLU Winogrande HellaSwag OpenBookQA LAMBADA SIQA PIQA ARC-e RACE TruthfulQA CrowS-Pairs
Metrics: validation loss training loss downstream accuracy
Baselines: cosine 10x decay constant LR inverse square root cosine D2Z WSD cyclic LR step decay
Builds on: AdamW maximal update parameterization warmup-stable-decay Chinchilla scaling laws SlimPajama
Closest papers by meaning
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.88)
- Scaling Optimal LR Across Token Horizons (ICLR 2025 · similarity 0.87)
- How to set AdamW's weight decay as you scale model and dataset size (ICML 2025 · similarity 0.86)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.86)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.85)
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.