AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Olmix: A Framework for Data Mixing Throughout LM Development

Mayee F Chen, Tyler Murray, David Heineman, Matt Jordan, Hannaneh Hajishirzi, Christopher Re, Luca Soldaini, Kyle Lo

ICML 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use proxy models of at least 15M parameters instead of 1M (ablation)OlmixBase swarm · DCLM 24 domains, 1B targetrank correlation with 1B model: 73 -> >89Improved
scale swarm size linearly with number of domains, K>=3(m+1) (ablation)OlmixBase · DCLM, m=6..24 domainsBPB error vs best mixImproved
use log-linear regression instead of LightGBM, Gaussian Process, BiMix or AutoScale (ablation)OlmixBase · DCLM 24 domainsregression fitImproved
include repetition constraints in mixture optimization (ablation)OlmixBase · data-constrained mixingproposed mixture weightsImproved
mix with OlmixBase-style mixture reuse instead of natural distributionFullMixtureReuse · Evolving domains, 5 updates (1B)% improvement in avg BPB over natural: 0 -> +11.6%Improved
use FullMixtureReuse instead of full recomputationmixing under evolving domains · Evolving domains, 5 updates (1B)% improvement / proxy runs: +12.2% -> +11.6%; 832 -> 216 runsMixed
use PartialMixtureReuse instead of FullMixtureReuse (ablation)mixture reuse · Evolving domains, 5 updates (1B)% improvement over natural: +11.6% -> +12%; 216 -> 272 runsImproved
use mixture reuse instead of swarm reuse (ablation)evolving domain mixing · Evolving domains, 5 updates (1B)% improvement over natural: +11.4% (268 runs) -> +11.6% (216 runs)Improved
use mixture reuse instead of full recomputation at c=1 for matched proxy-run budget (ablation)evolving domain mixing · Evolving domains, 5 updates (1B)% improvement over natural: +10.8% -> +11.6% / +12.0%Improved
train with PartialMixtureReuse mix instead of natural distribution1B model · Evolving domains, final domain settraining steps to reach natural final BPB: 61K -> ~20K steps (3.05x)Improved
reuse all unaffected domains when coupling with new domains is high (ablation)FullMixtureReuse · adding code data to web topicsperformance gap to full recomputationHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy