Research map / Continual learning / Continual fine-tuning of LLMs
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes data mixing laws, an exponential function of domain mixture proportions predicting validation loss, combined with nested scaling laws (steps, model size) to optimize pretraining mixtures from small-scale runs.
Problem
Pretraining data mixture proportions crucially affect LLM performance, but tuning them relies on heuristics, and evaluating candidate mixtures at target scale is prohibitively expensive.
Approach
Model validation loss as an exponential function of a linear combination of domain mixture proportions (data mixing law), fit on small-scale runs, and chain it with power-law scaling laws of training steps and model size to predict large-scale performance on unseen mixtures before training.
Key results
- The optimized RedPajama mixture for a 1B model trained on 100B tokens reaches the performance of the default mixture trained with 48% more steps (matching at 73% of steps).
- The exponential data mixing law (M4) achieves low MAE on three-domain experiments (e.g., ~0.003-0.037 validation MAE for M4 vs. 0.0050-0.0365 for M1), outperforming power-law and other candidate forms.
- Continual pretraining of Pythia-70M on Pile+Python: fitted laws predict the critical proportion that keeps Pile loss unchanged while improving the target domain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the default RedPajama mixture with the optimized mixture | 1B model trained on RedPajama · Pile validation | training steps for comparable performance: 0.73 of the original number of training steps; 48% more steps with the default mixture | Helped |
| Use M4 rather than a random guess for loss prediction | Data mixing law · GitHub validation | MAE: 0.8758 -> 0.0365 | Helped |
| Use M4 rather than a random guess for loss prediction | Data mixing law · Books3 validation | MAE: 0.1331 -> 0.0074 | Helped |
| Use M4 rather than a random guess for loss prediction | Data mixing law · Pile-CC validation | MAE: 0.1045 -> 0.0078 | Helped |
Surprising findings
- Optimal mixtures found at small scale (70M, 30k steps) are not consistently optimal at larger scales; rankings change with model size and training steps.
- Power-law forms were rejected for domain proportions because they blow up as proportions approach 0; an exponential form was used instead.
Limitations
- Domains are predefined open-source subsets rather than operationally defined domains, and the concept of a domain is not well-defined.
- Nested scaling-law predictions introduce error accumulation, with predictions tending to underestimate actual loss.
- Empirical findings without theoretical grounding; the joint law across steps, sizes, and mixtures is not derived.
Experimental setup
Datasets: The Pile RedPajama GitHub Books3 Pile-CC Wikipedia ArXiv StackExchange
Benchmarks: The Pile validation set domain-specific validation sets (GitHub, Pile-CC, Books3, and others)
Metrics: validation loss mean absolute error perplexity
Baselines: DoGE DoReMi default RedPajama mixture
Builds on: Chinchilla scaling laws DoReMi DoGE Pythia
Closest papers by meaning
- Aioli: A Unified Optimization Framework for Language Model Data Mixing (ICLR 2025 · similarity 0.92)
- RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.91)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.90)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.90)
- Data Mixing Can Induce Phase Transitions in Knowledge Acquisition (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.