Research map / Architectures and efficiency / RNNs and normalization
A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Multi-Power Law, an empirical law that predicts LLM pretraining loss curves for arbitrary learning rate schedules from a few fitted runs, and uses it to derive an optimized schedule that beats cosine and WSD.
Problem
Predicting the full pretraining loss curve of LLMs as a function of the entire learning rate schedule, which existing scaling laws (e.g., Chinchilla) do not account for, and finding better LR schedules without costly tuning.
Approach
A multi-power law that models loss as a power law in the cumulative LR sum minus a loss-reduction term that sums, over LR drops, a power-law saturating factor in the scaled LR sum; the law is fitted on 2-3 short runs and minimized over schedules to find improved LR schedules.
Key results
- On unseen schedules for 25M/100M/400M models, MPL achieves R^2 of 0.9975/0.9982/0.9971 versus 0.9904/0.9959/0.9962 for Momentum Law.
- The MPL-optimized schedule outperforms cosine on downstream tasks, e.g., LAMBADA 48.71 vs 46.54 and C3 50.30 vs 48.44 for 1B model.
- MPL reduces final-loss prediction error to under one-third of Chinchilla-style data scaling law while using about one-fifth the compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Multi-Power Law instead of Momentum Law for loss-curve prediction | Loss-curve prediction · Unseen schedules, 25M, 100M, and 400M | R²: 0.9904 -> 0.9975; 0.9959 -> 0.9982; 0.9962 -> 0.9971 | Helped |
| Optimize the learning-rate schedule using MPL instead of cosine | 400M model · LAMBADA | score: 46.54 -> 48.71 | Helped |
| Optimize the learning-rate schedule using MPL instead of cosine | 400M model · HellaSwag | score: 37.12 -> 37.74 | Helped |
| Optimize the learning-rate schedule using MPL instead of cosine | 400M model · PIQA | score: 65.13 -> 65.07 | Hurt |
| Optimize the learning-rate schedule using MPL instead of cosine | 400M model · ARC-E | score: 43.56 -> 44.09 | Helped |
| Optimize the learning-rate schedule using MPL instead of cosine | 400M model · C³ | score: 48.44 -> 50.30 | Helped |
| Optimize the learning-rate schedule using MPL instead of cosine | 400M model · RTE | score: 52.71 -> 53.79 | Helped |
| Remove the loss-reduction term (ablation) | MPL loss-curve fitting · 25M loss-curve fitting | R²: 0.8309 -> 0.9975 | Hurt |
| Replace the schedule-dependent loss-reduction function with a linear LR-reduction term (ablation) | MPL loss-curve fitting · 25M loss-curve fitting | R²: 0.9797 -> 0.9975 | Hurt |
| Set γ to 0 (ablation) | MPL loss-curve fitting · 25M loss-curve fitting | R²: 0.9961 -> 0.9975 | Hurt |
| Use step-based x=t-k instead of the learning-rate-scaled cumulative sum (ablation) | MPL loss-curve fitting · 25M loss-curve fitting | R²: 0.9921 -> 0.9975 | Hurt |
| Replace the power loss-reduction function with an exponential function (ablation) | MPL loss-curve fitting · 25M loss-curve fitting | R²: 0.9934 -> 0.9975 | Hurt |
Surprising findings
- The Momentum Law provably yields a collapsed schedule when optimized, whereas MPL recovers a WSD-like shape.
- Optimized schedule decays to near zero, and simply adjusting WSD's ending LR to near zero does not close the gap.
Limitations
- Derivation is empirical; theory covers only quadratic losses with SGD and does not fully match the functional form.
- Slight deviations from actual curves on long-horizon and high peak-LR runs; coefficients are batch-size and peak-LR dependent.
- Fitting with Adam is sensitive to initialization and lacks convergence guarantees.
Experimental setup
Datasets: data stream (pretraining corpus not named)
Benchmarks: LAMBADA HellaSwag PIQA ARC-easy C3 RTE
Metrics: R^2 MAE RMSE PredE WorstE validation loss
Baselines: Chinchilla data scaling law (CDSL) Momentum Law cosine schedule WSD WSDLD One-Power Law
Builds on: Chinchilla scaling law WSD schedule cosine schedule gradient flow approximation
Closest papers by meaning
- Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules (NeurIPS 2025 · similarity 0.91)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.90)
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.90)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.89)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.89)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.