Research map / Large language models / Language, safety and interpretability
Algorithmic progress in language models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Estimates how fast pre-training algorithms reduce compute needed for a fixed perplexity on WikiText-103/WikiText-2/Penn Treebank (2012-2023) using an augmented scaling law fit to ~231 models, finding compute-equivalent halving roughly every 8 months.
Problem
How much of recent language model progress comes from pre-training algorithmic improvements versus scaling compute and data, and how fast algorithms reduce compute needed for a fixed performance level.
Approach
Extends the Chinchilla-style scaling law with exponential 'effective parameter' and 'effective data' terms growing over time, so algorithmic progress appears as a constant rate of compute savings; fits this on a curated dataset of published perplexity results and decomposes gains with Shapley values.
Key results
- Effective compute required for fixed perplexity halves every 8.4 months (95% CI 4.5 to 14.3 months) under the preferred model.
- Shapley analysis attributes roughly 60-95% of performance gains to compute scaling and 5-40% to algorithms.
- The transformer architecture yields a median compute-equivalent gain of 7.2x (95% CI 3.3x to 45.7x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the transformer architecture rather than a non-transformer architecture | Language models evaluated on WikiText and Penn Treebank · Language modeling benchmarks | compute-equivalent gain: 7.2× [95% CI: 3.3×, 45.7×] | Helped |
| Add a benchmark vocabulary-size fixed effect (ablation) | Model 7 (preferred augmented scaling law) · WikiText-103, WikiText-2, Penn Treebank | effective compute doubling time: 8.4 months -> 8.0 months | No clear effect |
| Add a benchmark vocabulary-size fixed effect (ablation) | Model 7 (preferred augmented scaling law) · WikiText-103, WikiText-2, Penn Treebank | MSE loss: 0.0486 in both cases | No clear effect |
| Use alternative scaling-exponent formulations, including changing exponents over time (ablation) | Augmented scaling-law model · WikiText-103, WikiText-2, Penn Treebank | cross-validation R²: between 0.88 and 0.91 | No clear effect |
| Fit the model using only non-transformer models (ablation) | Augmented scaling-law model · Language modeling benchmarks | effective compute doubling time: 8.8 months; 95% confidence interval of -30.6 to 34.8 months | No clear effect |
| Switch from Kaplan scaling to Chinchilla scaling | Dense autoregressive transformer models · GPT-2 scale to PaLM scale compute budgets | compute-equivalent gain: between 1.75-fold (for GPT-2 scale models) and 4-fold (for PaLM-scale models) | Helped |
| Increase context length over time | Language-model evaluations · WikiText and Penn Treebank | loss reduction relative to algorithmic-progress loss reduction: 10-60% as large | Helped |
| Scale physical compute rather than rely only on algorithmic progress | Language models from 2014 to 2023 · Language modeling performance gains | share of performance gains: 60-95% | Helped |
Surprising findings
- Naive extrapolation implies algorithmic progress since 2014 equals about 22,000x more compute, yet this magnitude is not directly observed in the dataset.
- Non-transformer-only subset yields a doubling-time CI of -30.6 to 34.8 months, so the estimate is not statistically significant there.
Limitations
- Noisy and heterogeneous benchmark evaluations; data sparse before 2017 and many papers lack parameter counts.
- Relies on Chinchilla scaling law for non-transformer architectures; alternate exponents possible.
- Cannot attribute gains to specific innovations; confounds data quality with data efficiency and context length effects not fully controlled.
Experimental setup
Datasets: WikiText-103 WikiText-2 Penn Treebank
Benchmarks: WikiText-103 WikiText-2 Penn Treebank
Metrics: token-level perplexity doubling time of effective compute MSE test loss (leave-one-out cross-validation) Shapley value contribution compute-equivalent gain
Baselines: Kaplan scaling law Hernandez et al. 2020 compute-efficiency estimates Dorner 2021 RL sample efficiency
Builds on: Chinchilla scaling law Kaplan scaling laws Erdil and Besiroglu algorithmic progress in computer vision
Closest papers by meaning
- Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.90)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.89)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.89)
- Masked Structural Growth for 2x Faster Language Model Pre-training (ICLR 2024 · similarity 0.89)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.