AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

An empirical analysis of compute-optimal large language model training

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland and 10 more

NeurIPS 2022 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Scale model size and training tokens in equal proportion with compute (a=b≈0.5) instead of favoring model sizescaling law fit · Compute-optimal allocationexponent a for Nopt: 0.73 -> 0.50Improved
Fit IsoFLOP profiles at 9 FLOP budgets instead of minimum over training curves (ablation)scaling law estimation · Compute-optimal allocationexponent a: 0.50 -> 0.49No clear effect
Fit a parametric loss function with Huber loss instead of nonparametric approaches (ablation)scaling law estimation · Compute-optimal allocationexponent a: 0.50 -> 0.46Mixed
Train a 70B model on 1.4T tokens instead of 280B Gopher on 300B tokens at same FLOPsChinchilla · MMLU 5-shotaverage accuracy: +7.6%Improved
Train 70B model on more tokens vs Gopher 280BChinchilla · BIG-benchaverage accuracy: 54.4% -> 65.1%Improved
Train 70B model on more tokens vs Gopher 280BChinchilla · Wikitext103perplexity: 7.75 -> 7.16Improved
Train 70B model on more tokens vs Gopher 280BChinchilla · LAMBADA zero-shotaccuracy: 74.5 -> 77.4Improved
Train 70B model on more tokens vs Gopher 280BChinchilla · RACE-h few-shotaccuracy: 71.6 -> 82.3Improved
Train 70B model on more tokens vs Gopher 280BChinchilla · RACE-m few-shotaccuracy: 75.1 -> 86.8Improved
Train 70B model on more tokens vs Gopher 280BChinchilla · Winogrande zero-shotaccuracy: 70.1% -> 74.9%Improved
Train 70B model on more tokens vs Gopher 280BChinchilla · PIQA zero-shotaccuracy: 81.8% -> 81.8%No clear effect
Train 70B model on more tokens vs Gopher 280BChinchilla · BoolQ zero-shotaccuracy: 79.3% -> 83.7%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy