Research map / Large language models / Instruction and fine-tuning
MergeBench: A Benchmark for Merging Domain-Specialized LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MergeBench, an open benchmark that finetunes 2B–9B Llama-3 and Gemma-2 models on five domains and evaluates eight model merging methods on multi-task performance, forgetting, and runtime.
Problem
Existing model merging evaluations use small models (GPT-2, RoBERTa, mT5) and narrow NLP tasks, leaving unclear how merging methods perform on large, domain-specialized LLMs.
Approach
A standardized benchmark that builds 40 specialized models from recent open LLM families (Llama-3.2/3.1, Gemma-2 at 2B–9B, pretrained and instruct versions) across five post-training domains, and evaluates merging on multi-task performance, retention of base-model knowledge, and wall-clock runtime.
Key results
- Merged models on 8B/9B bases recover over 90% of finetuned normalized performance, versus about 80% at 2B/3B; Localize-and-Stitch achieves the highest overall normalized performance (89.7%).
- Model Soup is the cheapest method (no tuning), while TIES and DARE have the longest total runtime due to grid-search validation.
- Merged models retain base knowledge better than multi-task models; sparsification (TIES, Dataless L&S) and smaller scaling coefficients reduce forgetting, while DARE retains less.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use larger pretrained base models (8B/9B rather than 2B/3B) | model merging methods · MergeBench five task categories | normalized multi-task performance | Helped |
| Use instruction-tuned rather than pretrained base models | model merging methods · MergeBench five task categories | normalized multi-task performance | Helped |
| Tune merging coefficients to use smaller scaling coefficients | merged models · base-model generalization benchmarks | normalized generalization performance | Helped |
| Apply sparsification to task-vector updates | merged models · base-model generalization benchmarks | normalized generalization performance | Helped |
| Use random task-vector dropout rather than targeted sparsification | DARE · base-model generalization benchmarks | normalized generalization performance | Hurt |
| Use multi-task training instead of model merging | multi-task models · in-domain tasks | in-domain performance | Helped |
| Use Localize-and-Stitch rather than less effective merging methods | model merging methods · MergeBench five task categories | normalized multi-task performance | Helped |
| Use DARE random dropout rather than other merging approaches | model merging methods · larger models | normalized multi-task performance | Hurt |
Surprising findings
- Validation (hyperparameter tuning) cost can dominate total merging runtime in real-world use.
- Merging Gemma models often yields stronger results than merging Llama models of similar size.
Limitations
- Only merges models finetuned from the same initialization within the same model family; dense models only, no MoE.
- Multi-task models still outperform merged models on in-domain performance when tasks are non-conflicting and balanced.
- Insights are empirical; no theoretical analysis, and merging hyperparameter tuning remains costly and trial-and-error.
Experimental setup
Datasets: TULU-3 persona IF DART-Math NuminaMath-TIR Aya Magicoder WildGuardMix WildJailbreak IFEval GSM8k MATH M_MMLU M_ARC
Benchmarks: IFEval GSM8k MATH M_MMLU M_ARC M_Hellaswag HumanEval+ MBPP+ WildGuardTest HarmBench DoAnythingNow XSTest
Metrics: normalized performance accuracy pass@1 exact match refusal rate (RTA) wall-clock runtime
Baselines: Model Soup Task Arithmetic Fisher Merging RegMean TIES DARE Consensus TA Localize-and-Stitch
Builds on: Model Soup Task Arithmetic Fisher Merging RegMean TIES-Merging DARE Localize-and-Stitch Consensus TA GRPO
Closest papers by meaning
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.94)
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.92)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.92)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.91)
- Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.91)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.