Research map / Large language models / AI and society
Position: The Most Expensive Part of an LLM *should* be its Training Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that LLM training data labor should be the most expensive part of producing an LLM, estimating conservative replacement costs for 64 language models' datasets versus their training costs.
Problem
Training data for LLMs is mostly scraped uncompensated from the web, so the human labor behind it is unaccounted for in LLM development costs; the paper asks how much compensation training data creators would be owed.
Approach
Applies a replacement-cost valuation to training datasets (words written at 30 wpm times a conservative $3.85/hour median minimum wage) and compares it with hardware, energy, and engineering training costs from prior scaling-trends analysis for 64 language models (2016-2024), finding dataset costs 10-1000x training costs.
Key results
- Training dataset replacement costs are at least an order of magnitude larger than training costs for every one of the 64 models studied, e.g. 300x for GPT-4 and 6000x for DeepSeek-V3.
- Under the conservative valuation, many recent datasets are valued above $10 billion USD.
- For 8 of 10 companies examined, estimated training data cost exceeds 10% of annual revenue; for 3 of 10 it exceeds total annual revenue.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate dataset production costs using replacement-cost labor estimates | 64 language models released between 2016 and 2024 · LLM training datasets versus model training | dataset cost / training cost: 10-1000 times | Helped |
| Estimate dataset production costs using replacement-cost labor estimates | GPT-4 · GPT-4 training dataset versus training run | dataset cost / training cost: 300× | Helped |
| Estimate dataset production costs using replacement-cost labor estimates | DeepSeek-V3 · DeepSeek-V3 training dataset versus training run | dataset cost / training cost: 6000× | Helped |
Surprising findings
- Even under deliberately conservative wage and writing-speed assumptions, dataset costs exceed training costs by orders of magnitude.
- The Encyclopedia Britannica's actual production cost was about 400x higher than the paper's estimate, suggesting estimates are underestimates.
Limitations
- Ignores text quality and content type, treating all words equally at a single writing speed and wage.
- Relies on training-cost estimates from prior work and on dataset sizes that providers often do not disclose.
- Argues from a normative position; counterarguments such as fair use and innovation concerns are discussed but not resolved.
Experimental setup
Datasets: training datasets of 64 language models released 2016-2024 (sizes from prior scaling-trends database)
Metrics: estimated dataset replacement cost (USD) estimated training cost (USD) dataset cost as fraction of annual revenue
Builds on: Epoch AI training cost estimates
Closest papers by meaning
- A Theory of Training Profit-Optimal LLMs (NeurIPS 2026 · similarity 0.89)
- Fairshare Data Pricing via Data Valuation for Large Language Models (NeurIPS 2025 · similarity 0.84)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.83)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.83)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.83)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.