Research map / Large language models / Language, safety and interpretability
Olmix: A Framework for Data Mixing Throughout LM Development
Mayee F Chen, Tyler Murray, David Heineman, Matt Jordan, Hannaneh Hajishirzi, Christopher Re, Luca Soldaini, Kyle Lo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use proxy models of at least 15M parameters instead of 1M (ablation) | OlmixBase swarm · DCLM 24 domains, 1B target | rank correlation with 1B model: 73 -> >89 | Improved |
| scale swarm size linearly with number of domains, K>=3(m+1) (ablation) | OlmixBase · DCLM, m=6..24 domains | BPB error vs best mix | Improved |
| use log-linear regression instead of LightGBM, Gaussian Process, BiMix or AutoScale (ablation) | OlmixBase · DCLM 24 domains | regression fit | Improved |
| include repetition constraints in mixture optimization (ablation) | OlmixBase · data-constrained mixing | proposed mixture weights | Improved |
| mix with OlmixBase-style mixture reuse instead of natural distribution | FullMixtureReuse · Evolving domains, 5 updates (1B) | % improvement in avg BPB over natural: 0 -> +11.6% | Improved |
| use FullMixtureReuse instead of full recomputation | mixing under evolving domains · Evolving domains, 5 updates (1B) | % improvement / proxy runs: +12.2% -> +11.6%; 832 -> 216 runs | Mixed |
| use PartialMixtureReuse instead of FullMixtureReuse (ablation) | mixture reuse · Evolving domains, 5 updates (1B) | % improvement over natural: +11.6% -> +12%; 216 -> 272 runs | Improved |
| use mixture reuse instead of swarm reuse (ablation) | evolving domain mixing · Evolving domains, 5 updates (1B) | % improvement over natural: +11.4% (268 runs) -> +11.6% (216 runs) | Improved |
| use mixture reuse instead of full recomputation at c=1 for matched proxy-run budget (ablation) | evolving domain mixing · Evolving domains, 5 updates (1B) | % improvement over natural: +10.8% -> +11.6% / +12.0% | Improved |
| train with PartialMixtureReuse mix instead of natural distribution | 1B model · Evolving domains, final domain set | training steps to reach natural final BPB: 61K -> ~20K steps (3.05x) | Improved |
| reuse all unaffected domains when coupling with new domains is high (ablation) | FullMixtureReuse · adding code data to web topics | performance gap to full recomputation | Hurt |
Closest papers by meaning
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.90)
- Aioli: A Unified Optimization Framework for Language Model Data Mixing (ICLR 2025 · similarity 0.90)
- RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.89)
- Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.89)
- Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning (ICML 2025 · similarity 0.89)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
