Research map / Label-efficient and robust learning / Domain adaptation and generalization
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, Adams Wei Yu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reweight Pile domains with DoReMi (280M proxy) instead of default weights | 8B LM · The Pile, avg one-shot accuracy | avg exact match: 20.03 -> 26.56 | Improved |
| reweight Pile domains with DoReMi (280M proxy) | 8B LM · The Pile val | avg log-ppl: 1.64 -> 1.40 | Improved |
| increase proxy model size from 70M to 280M (ablation) | DoReMi with 8B main model · The Pile, avg one-shot accuracy | avg exact match: 21.11 -> 26.56 | Improved |
| increase proxy model size from 280M to 1B (ablation) | DoReMi with 8B main model · The Pile, avg one-shot accuracy | avg exact match: 26.56 -> 22.35 | Hurt |
| use excess loss instead of only proxy loss (hardest domains) (ablation) | DoReMi 280M · The Pile val | avg log-ppl: 2.62 -> 2.13 | Improved |
| use excess loss instead of only negative reference loss (easiest domains) (ablation) | DoReMi 280M · The Pile val | avg log-ppl: 4.18 -> 2.13 | Improved |
| match proxy and main size, DoReMi at 280M | 280M LM · The Pile val | avg log-ppl: 2.32 -> 2.13 | Improved |
| match proxy and main size, DoReMi at 510M | 510M LM · The Pile val | worst-case log-ppl: 2.14 -> 2.14 | No clear effect |
| use trained 1B proxy model directly vs main model with DoReMi weights (ablation) | 1B LM · The Pile val | avg log-ppl: 2.02 vs 1.83 | Hurt |
| iterate DoReMi to round 2 on GLaM dataset (ablation) | 8B LM · GLaM dataset downstream | avg one-shot accuracy | Improved |
Closest papers by meaning
- Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning (ICML 2025 · similarity 0.89)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.89)
- GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining (NeurIPS 2025 · similarity 0.89)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.88)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.88)
- TANDEM: Bi-Level Data Mixture Optimization with Twin Networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
