Research map / Architectures and efficiency / RNNs and normalization
Model Merging Scaling Laws in Large Language Models
Yuanyi Wang, Yanggan Gu, Yiming Zhang, Qi Zhou, Zhaoyi Yan, Congkai Xie, Xinyao Wang, Jianbo Yuan, Hongxia Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a larger base model (0.5B to 32B) when merging nine experts | Average merge · In-domain macro CE, k=9 | cross-entropy: 0.739 -> 0.430 | Improved |
| merge more experts (k=1 to k=9) at 0.5B | Average merge · In-domain macro CE | cross-entropy: 0.816 -> 0.739 | Improved |
| merge more experts (k=1 to k=9) at 32B | Average merge · In-domain macro CE | cross-entropy: 0.493 -> 0.430 | Improved |
| use more merged experts beyond k=6 | floor+tail law · Fractional return of CE improvement | fraction of gain: 85% by k=5, 90% by k=6 | No clear effect |
| compare merge methods at larger k (TA, TIES, DARE vs Average) | 32B merging · Macro CE across nine domains | cross-entropy: differences <2% by k=8 | No clear effect |
| restrict candidate pool from M=8 to M=7 domains (ablation) | DARE cross-domain merging · Fitted tail A(N) | tail amplitude | Mixed |
| fit the law on only k in {1,2,4} to forecast the full curve | floor+tail law · Forecast of k=1..9 curve | forecast error: k* around 5-6 | Improved |
| increase number of merged experts k in DARE order permutations | DARE at 32B · Across-order CE spread | worst-best CE spread: 0.086 -> 0.015 | Improved |
| increase k to 9 in order permutation study at 72B | DARE · Relative range reduction | range reduction: 24% (0.5B), 32% (32B), 34% (72B) | Improved |
| apply the same law to LLaMA backbones | floor+tail law · Cross-domain macro CE | R2: R2=0.999 and R2=0.995 | Improved |
| evaluate downstream tasks instead of CE | merged models on LLaMA · Downstream benchmark scores | normalized score | Mixed |
Closest papers by meaning
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.90)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.90)
- A Tale of Tails: Model Collapse as a Change of Scaling Laws (ICML 2024 · similarity 0.89)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
