Research map / Architectures and efficiency / Pruning and efficient tuning
An empirical analysis of compute-optimal large language model training
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland and 10 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale model size and training tokens in equal proportion with compute (a=b≈0.5) instead of favoring model size | scaling law fit · Compute-optimal allocation | exponent a for Nopt: 0.73 -> 0.50 | Improved |
| Fit IsoFLOP profiles at 9 FLOP budgets instead of minimum over training curves (ablation) | scaling law estimation · Compute-optimal allocation | exponent a: 0.50 -> 0.49 | No clear effect |
| Fit a parametric loss function with Huber loss instead of nonparametric approaches (ablation) | scaling law estimation · Compute-optimal allocation | exponent a: 0.50 -> 0.46 | Mixed |
| Train a 70B model on 1.4T tokens instead of 280B Gopher on 300B tokens at same FLOPs | Chinchilla · MMLU 5-shot | average accuracy: +7.6% | Improved |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · BIG-bench | average accuracy: 54.4% -> 65.1% | Improved |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · Wikitext103 | perplexity: 7.75 -> 7.16 | Improved |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · LAMBADA zero-shot | accuracy: 74.5 -> 77.4 | Improved |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · RACE-h few-shot | accuracy: 71.6 -> 82.3 | Improved |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · RACE-m few-shot | accuracy: 75.1 -> 86.8 | Improved |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · Winogrande zero-shot | accuracy: 70.1% -> 74.9% | Improved |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · PIQA zero-shot | accuracy: 81.8% -> 81.8% | No clear effect |
| Train 70B model on more tokens vs Gopher 280B | Chinchilla · BoolQ zero-shot | accuracy: 79.3% -> 83.7% | Improved |
Closest papers by meaning
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.91)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.90)
- Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · similarity 0.90)
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (ICML 2024 · similarity 0.90)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
