Research map / Large language models / Instruction and fine-tuning
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
jiapeng wang, Changxin Tian, Kunlong Chen, Ziqi Liu, Jiaxin Mao, Xin Zhao, Zhiqiang Zhang, JUN ZHOU
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MergeMix-derived mixture instead of manually tuned ratios | 8B MoE mid-training · Mid-training average of 5 domains | average score: 57.9 -> 58.4 | Improved |
| use MergeMix mixture instead of uniform sampling | 8B MoE mid-training · Mid-training average | average score: 54.2 -> 58.4 | Improved |
| use MergeMix instead of Adapted RegMix | 8B MoE mid-training · Mid-training average | average score: 58.1 -> 58.4 | Improved |
| replace manual mixture search by merging-based search | mixture tuning · search cost | cost in tokens: 2T tokens -> 20B tokens (100x) | Improved |
| weight interpolation of math/code experts as proxy for data mixing | math and code experts · math/code ratio sweep | Spearman rho: 0.82 (math), 0.78 (code) | Improved |
| transfer the mixture found on 8B proxy to 16B model | 16B MoE mid-training · cross-scale transfer | average score | Improved |
| re-optimize mixture at 50% of training (dynamic) vs static (ablation) | MergeMix · mid-training average | average score | No clear effect |
| MergeMix fine-grained weights instead of uniform within domains (ablation) | hierarchical MergeMix · 16 sub-clusters | average score: 57.6 -> 58.2 | Improved |
| MergeMix fine-grained weights instead of manual fine-grained (ablation) | hierarchical MergeMix · 16 sub-clusters | average score: 58.4 -> 58.2 | No clear effect |
| fully automated MergeMix hierarchy instead of fully manual | mid-training mixture · 16 sub-clusters | average score: 57.9 -> 58.2 | Improved |
| MergeMix mixture vs uni-domain and aggressive specialization mixtures | math and code targeting · math and code benchmarks | accuracy: math 60.14, 60.96 vs MergeMix 61.48 | Improved |
Closest papers by meaning
- Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.93)
- RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.91)
- Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging (NeurIPS 2025 · similarity 0.91)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.90)
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.90)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
