AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Model Merging Scaling Laws in Large Language Models

Yuanyi Wang, Yanggan Gu, Yiming Zhang, Qi Zhou, Zhaoyi Yan, Congkai Xie, Xinyao Wang, Jianbo Yuan, Hongxia Yang

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use a larger base model (0.5B to 32B) when merging nine expertsAverage merge · In-domain macro CE, k=9cross-entropy: 0.739 -> 0.430Improved
merge more experts (k=1 to k=9) at 0.5BAverage merge · In-domain macro CEcross-entropy: 0.816 -> 0.739Improved
merge more experts (k=1 to k=9) at 32BAverage merge · In-domain macro CEcross-entropy: 0.493 -> 0.430Improved
use more merged experts beyond k=6floor+tail law · Fractional return of CE improvementfraction of gain: 85% by k=5, 90% by k=6No clear effect
compare merge methods at larger k (TA, TIES, DARE vs Average)32B merging · Macro CE across nine domainscross-entropy: differences <2% by k=8No clear effect
restrict candidate pool from M=8 to M=7 domains (ablation)DARE cross-domain merging · Fitted tail A(N)tail amplitudeMixed
fit the law on only k in {1,2,4} to forecast the full curvefloor+tail law · Forecast of k=1..9 curveforecast error: k* around 5-6Improved
increase number of merged experts k in DARE order permutationsDARE at 32B · Across-order CE spreadworst-best CE spread: 0.086 -> 0.015Improved
increase k to 9 in order permutation study at 72BDARE · Relative range reductionrange reduction: 24% (0.5B), 32% (32B), 34% (72B)Improved
apply the same law to LLaMA backbonesfloor+tail law · Cross-domain macro CER2: R2=0.999 and R2=0.995Improved
evaluate downstream tasks instead of CEmerged models on LLaMA · Downstream benchmark scoresnormalized scoreMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy