Research map / Large language models / Language, safety and interpretability
Position: Will we run out of data? Limits of LLM scaling based on human-generated data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Projects LLM training dataset growth against an estimated stock of public human-generated text, forecasting that data will be fully utilized around 2026-2032 and surveying ways to circumvent the bottleneck.
Problem
Whether the finite stock of public human-generated text data can sustain continued scaling of LLM training datasets and compute.
Approach
Combines an estimated stock of public human text (indexed web via Google index size and Common Crawl bytes per page, plus an internet-user-based model) with a mixture of historical and compute-based projections of training dataset size, adjusted for quality filtering and multi-epoch training, to predict when the stock is exhausted.
Key results
- Median year public human text stock is fully utilized is 2028, with exhaustion very likely by 2032; models would then train at ~5e28 FLOP.
- Indexed web stock estimated at ~510T tokens (95% CI 130T-2100T); whole web (including deep web) ~3100T tokens.
- Overtraining by 5x moves the data bottleneck about one year earlier, at ~6e27 FLOP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Assume frontier models are overtrained by 5x starting from 2025 | Compute-based data-usage projection · Public human text data availability | training compute at bottleneck: ~5e28 FLOP -> ~6e27 FLOP | Hurt |
| Add access to non-public platform and messaging data | Public human text data stock projection · Public human text data availability | bottleneck timing: about a year and a half | Helped |
| Use data from images and video in addition to text | Public human text data stock projection · Data availability | bottleneck timing | No clear effect |
| Use synthetic data for training | Language models · Mathematics, programming, and games | performance | Helped |
Surprising findings
- Google's index size appears roughly constant over the past decade despite growth in internet users and IP traffic.
- Non-public data (social media and messaging) could be on the order of a quadrillion tokens, but would delay the bottleneck only about a year and a half.
Limitations
- Stock estimates are rough and span wide confidence intervals; quality adjustments rely on vague working definitions.
- Assumes compute-optimal training and constant data production per user; growth-rate assumptions are uncertain.
- Qualitative assessment of synthetic data, transfer learning, and non-public data viability.
Experimental setup
Datasets: Common Crawl RefinedWeb C4 RedPajama The Pile Pushshift Reddit
Metrics: median exhaustion year stock size in tokens training compute at exhaustion (FLOP)
Builds on: Chinchilla scaling law Villalobos-style notable models database (Epoch) Muennighoff et al. data-constrained scaling Common Crawl RefinedWeb
Closest papers by meaning
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Position: Enough of Scaling LLMs! Lets Focus on Downscaling (ICML 2025 · similarity 0.88)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
- Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective (ICLR 2025 · similarity 0.87)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.