Research map / Large language models / Language, safety and interpretability
Scaling Optimal LR Across Token Horizons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study showing the optimal learning rate for LLM pretraining decreases with token horizon following a power law, enabling transfer of LR across token horizons, and suggesting LLaMA-1 used too high an LR.
Problem
Hyperparameter transfer across model sizes (muP) is well studied, but how the optimal learning rate should change with the training token horizon has not been studied; practitioners need LR for long runs inferred from short ones.
Approach
Shows that optimal LR decreases with token horizon as a power law LR*(D)=B D^-beta, fits this from short-horizon sweeps to predict long-horizon optimal LR, extends it jointly with model size as C N^-alpha D^-beta, and applies it to argue LLaMA-1's 3e-4 LR was too high.
Key results
- The optimal LR decreases with token horizon across model sizes (50M to 2.7B) and under muP; LR*(D)=B D^-beta fits with R^2 0.96-0.99.
- Simulated transfer for a 50M model predicts LR at 800B tokens with 10-15% relative error, versus >250% when reusing the 100B-optimal LR.
- Extrapolating the scaling law gives an optimal LR of 1.15e-4 at 1T tokens for LLaMA-1 7B, so LLaMA-1's 3e-4 is too large by a factor above 2.5; estimated loss penalty upper bound 0.027.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the token horizon while choosing the optimal learning rate for each horizon (ablation) | GPT-3 parameterization across model sizes · LLM training, 25B–800B tokens | optimal learning rate | Helped |
| Use a power-law scaling fit to transfer the optimal LR from shorter to longer horizons | 50m model · 50m model, up to 800B tokens | relative error in predicted optimal LR: 10-15\% vs >250\% | Helped |
| Use muP parameterization instead of the standard transformer parameterization (ablation) | 50m model · 25B, 50B, and 100B tokens | optimal learning rate transfer across horizons | No clear effect |
| Increase batch size from 0.5m to 1m tokens (ablation) | 1.3B model · 25B and 50B tokens | optimal learning rate | Helped |
| Fit the horizon scaling law using the 25B, 50B, and 100B measurements (ablation) | 50m model · 200B, 400B, and 800B tokens | prediction ratio: 0.873, 0.894, 1.14 | Mixed |
| Train at the Llama-1 learning rate rather than the estimated optimal learning rate | Llama-1 7B · 1T tokens | optimality of learning rate: \scinot{1.15}{-4} vs \scinot{3}{-4}; factor of >2.5 | Hurt |
| Use a 1m-token batch instead of a 0.5m-token batch (ablation) | 1.3B model · 25B and 50B tokens | dependence of optimal LR on token horizon | No clear effect |
| Train for longer horizons over a fixed set of 25B unique tokens (ablation) | 50m model · 25B, 50B, and 100B total tokens | optimal learning rate | Helped |
Surprising findings
- The optimal LR still decreases with token horizon under muP parametrization, so muP does not transfer LR across token horizons.
- Repeating the 50M-model sweep with the same token budget but multiple epochs over 25B unique tokens shows the optimal LR depends on total token horizon, not just unique tokens.
Limitations
- Only extends scaling laws to roughly 800B tokens, far shorter than many SOTA training runs.
- Model-size dependence only preliminary; joint scaling law fit on 760M-2.7B models with a single 7B held-out point.
- Beta differs for the smallest models (50M, 125M), so the law is valid only for models of roughly 760M parameters and above.
Experimental setup
Datasets: RefinedWeb
Metrics: final validation loss R^2 relative error of predicted optimal LR bootstrapped standard deviation
Baselines: muP parametrization standard parametrization (GPT-3 recipe) LLaMA-1 learning rate
Builds on: muP GPT-3 scaling laws Chinchilla scaling laws Megatron-LM
Closest papers by meaning
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.91)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.88)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.88)
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.87)
- Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs (ICLR 2025 · similarity 0.87)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.