Research map / Large language models / Reasoning and chain of thought
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Modifies Chinchilla scaling laws to include inference FLOPs and real-world dollar costs, finding inference-heavy deployments favor smaller, longer-trained models, and validates with 47 MPT-style models up to 10,000 tokens/parameter.
Problem
Chinchilla scaling laws optimize only training compute and ignore inference cost, so they can prescribe suboptimal parameter and token counts when models serve large inference demand.
Approach
Fix the Chinchilla parametric loss at a target quality and minimize total training plus lifetime inference FLOPs (or real-world cost with MFU and per-FLOP prices), solving numerically for optimal parameters and tokens; also refit Chinchilla coefficients on extreme token-ratio runs to show overestimated data benefits.
Key results
- For a 30B-Chinchilla-quality model with 10^13 inference tokens, training a 13.6B model on 2.84x the data cuts total FLOPs by 28%.
- For a 30B-Chinchilla-quality model with 1.5B inference requests, a cost-optimal 16B model trained on 3.35T tokens reduces cost by 17%.
- Fitting Chinchilla coefficients only on data up to 100 tokens/parameter gives beta=0.13 and alpha=0.08, versus alpha=0.18 and beta=0.24 on all data, indicating flatter scaling at extreme ratios; model loss keeps decreasing up to 10,000 tokens/parameter for the 150M model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase tokens per parameter | MPT models from 150M to 6B parameters · Training sweep, up to 10,000 tokens per parameter for 150M models | training loss | Helped |
| Increase tokens per parameter | MPT models from 150M to 6B parameters · Evaluation Gauntlet | Gauntlet Average | Helped |
| Optimize model size and training tokens for total training and inference FLOPs | Chinchilla-style models · Compute-optimal configurations in the selected table | FLOP reduction: 9.1%; 2.6%; 8.5%; 16%; 12% | Helped |
| Optimize model size and training tokens for real-world costs, including hardware utilization | Chinchilla-style models · Cost-optimal configurations in the selected table | Cost savings: 50%; 34%; 49%; 58%; 54% | Helped |
| Fit scaling-law coefficients using only runs with no more than 100 tokens per parameter (ablation) | Chinchilla parametric scaling law · Extreme token-per-parameter training regime | accuracy of estimated loss improvement from additional training data | Hurt |
| Include runs up to 250 tokens per parameter when fitting scaling-law coefficients (ablation) | Chinchilla parametric scaling law · Extreme token-per-parameter training regime | scaling-curve slope: 0.13; 0.16 | Mixed |
| Include runs up to 500 tokens per parameter when fitting scaling-law coefficients (ablation) | Chinchilla parametric scaling law · Extreme token-per-parameter training regime | scaling-curve slope: 0.13; 0.16 | Mixed |
| Fit scaling-law coefficients using all training data, including extreme-ratio runs (ablation) | Chinchilla parametric scaling law · Extreme token-per-parameter training regime | scaling-curve slope: 0.18; 0.24 | Helped |
Surprising findings
- Model loss continues to decrease up to 10,000 tokens per parameter with no evidence of a saturation point for the 150M model.
- None of the fitted parametric curves fit the 150M long-ratio runs well, suggesting the Chinchilla parametric form is not flexible enough.
Limitations
- Experiments limited to 47 models up to 6B parameters, far fewer than Chinchilla's 400+ runs, so fitted coefficients have wide uncertainty.
- Assumes inference demand is independent of model size and known before training; ignores latency and differences in demand for smaller models.
- Long-ratio results only tested up to 1,000 tokens/parameter for larger models, so behavior beyond is unverified.
Experimental setup
Datasets: web text and code (Dataset consists of trillions of tokens) LMSYS-Chat (inference request statistics)
Benchmarks: Evaluation Gauntlet (MMLU, ARC, HellaSwag, GSM8k, SQuAD, BoolQ, and others)
Metrics: pre-training cross-entropy loss Gauntlet Average accuracy total FLOPs total dollar cost
Baselines: Chinchilla-style compute-optimal models
Builds on: Chinchilla scaling laws MPT Evaluation Gauntlet Lion optimizer
Closest papers by meaning
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.92)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.92)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.91)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.91)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.90)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.