Research map / Architectures and efficiency / RNNs and normalization
A Hitchhiker's Guide to Scaling Law Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta-analysis of scaling-law estimation across 485 published pretrained models from 40+ scaled families, deriving practical best practices (use intermediate checkpoints, more small models, cross-family parameter sharing).
Problem
How to best estimate and interpret scaling laws for LLM pretraining so that small or partially trained models reliably predict the loss of large target models.
Approach
Collects a large public dataset of losses and downstream evaluations across many model families, then systematically studies which models, checkpoints, and parameters to use when fitting the standard power-law scaling form, yielding practical guidelines (fit on intermediate checkpoints, drop the first ~10B tokens, train more small models, share size-scaling parameters across families).
Key results
- Fitting scaling laws on intermediate checkpoints (not just final losses) substantially improves accuracy; final loss can often be predicted from roughly the first third of training, with the first ~10B tokens best dropped.
- Baselines (best model or most-trained model) suffer about 18% ARE on average, whereas fitted scaling laws typically reach around 4% ARE, the effect size considered meaningful.
- In OLMo, a scaling law fitted from 1B checkpoints predicts a 7B model's loss with under 1% error when using a single model from another family's shared size parameters; OPT extrapolation errors reach 37% from 8.7B to 175B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fit scaling laws with intermediate checkpoints rather than final losses alone | Scaling law estimation · Scaled model families | ARE | Helped |
| Ignore the first 10B tokens seen when fitting (ablation) | Scaling laws for OPT and Pythia · OPT and Pythia | ARE: exceeds 15% -> 4-10% | Helped |
| Use a single model from the target family with scaling parameters from another family | Cross-family scaling law prediction · OLMO 7B from 1B checkpoints | ARE: less than 1% error | Helped |
| Use a larger source model when extrapolating to OPT 175B | Cross-family scaling law prediction · OPT 175B | ARE: 37%, 25% and 15% when extrapolating from 8.7B, 13B and 30B | Helped |
| Fit a scaling law instead of using best-performance or most-trained-model baselines | Scaling law estimation · All scaled families studied | ARE: baselines ... more than 15% error; 18% ARE on average | Helped |
| Train more models, including smaller models, rather than relying on fewer models | Scaling law estimation · GPT-3 and other scaled families | ARE | Helped |
| Add a larger source model to reduce extrapolation distance | Scaling law estimation · Scaled model families | ARE | Mixed |
| Use training checkpoints while discarding the earliest part of training (ablation) | Scaling law estimation · Scaled model families | ARE | Helped |
| Predict from smaller models when the target is much larger | Scaling law estimation · GPT, Gopher and OPT | ARE: less than 10% error | Helped |
| Remove a poorly predictive model based on cross-validation (ablation) | Scaling law estimation · Scaled model families | ARE: 58% | Hurt |
| Estimate a target model's final loss from roughly the first 30% of its training run | Scaling law estimation · Target model training trajectory | ARE: 30% | Helped |
| Train models closer in size to the target | Scaling law estimation · Scaled model families | ARE | Helped |
Surprising findings
- Adding larger models to the fitting set can sometimes hurt accuracy due to seed variance (e.g., Pythia 2.8B), and cross-validation fails to detect such models.
- Scaling laws appear to have fewer degrees of freedom than assumed: 3 principal components explain 99.49% of variance across the 5 parameters, with linear relations between A and alpha and between B and beta.
Limitations
- ARE does not distinguish over- from under-estimation of performance.
- Difficult to aggregate information across model families because published families span incomparable scales and ranges.
- Functional form assumed to be fixed and ignores learning-rate schedule effects.
Experimental setup
Datasets: aggregated training losses and downstream evals from 485 pretrained models across 40+ scaled families
Benchmarks: held-out loss of 30%-maximal-token target checkpoints across scaled families
Metrics: absolute relative error (ARE) Huber loss (robustness check)
Baselines: best-performance baseline most-trained-model baseline
Builds on: Chinchilla scaling laws (Hoffmann et al. 2022) Kaplan et al. 2020 scaling laws
Closest papers by meaning
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.95)
- Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets (NeurIPS 2025 · similarity 0.93)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.93)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.92)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.92)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.