Research map / Architectures and efficiency / Pruning and efficient tuning
Scalable Model Merging with Progressive Layer-wise Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProDistill, a few-shot model merging method that learns element-wise merging coefficients via progressive layer-wise teacher-student feature distillation with dual inputs, and proves data-agnostic merging can be arbitrarily bad in the worst case.
Problem
Model merging of fine-tuned models degrades performance, especially with few or no data; the paper asks whether domain-specific data is necessary and how to use few-shot unlabeled data to merge models efficiently, including large (>10B) LLMs.
Approach
Frame merging as teacher-student distillation: learn element-wise merging coefficients by matching internal activations of the merged model to each fine-tuned model, trained layer by layer with dual cached inputs (merged-model and fine-tuned activations), which reduces memory and improves performance over end-to-end training.
Key results
- ProDistill reaches 86.04% average on eight ViT-B-32 vision tasks with 64 shots, about 6.14 points above the best baseline (Table 1 and text).
- On RoBERTa NLU tasks ProDistill averages 0.7641 vs 0.6980 for RegMean-level baselines, a 6.61-point gain claimed in the abstract.
- On WizardMath-13B + Llama-2-13b-code-alpaca merging, ProDistill gets 0.8288 normalized average vs 0.7868 for TIES-Merging.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ProDistill instead of Localize-and-Stich | ViT-B-32 merging · Vision tasks (8-task Avg) | accuracy: 79.90 -> 86.04 | Helped |
| use ProDistill instead of RegMean | RoBERTa model merging · GLUE tasks (Avg) | average score: 0.6980 -> 0.7641 | Helped |
| use ProDistill instead of Task Arithmetic | WizardMath-13B + Llama-2-13b-code-alpaca merging · GSM8K | zero-shot accuracy: 0.6467 -> 0.6279 | Hurt |
| use ProDistill instead of Task Arithmetic | WizardMath-13B + Llama-2-13b-code-alpaca merging · MATH | zero-shot accuracy: 0.1462 -> 0.1424 | Hurt |
| use ProDistill instead of TIES-Merging | WizardMath-13B + Llama-2-13b-code-alpaca merging · HumanEval | pass@1: 0.0976 -> 0.1280 | Helped |
| use ProDistill instead of TIES-Merging | WizardMath-13B + Llama-2-13b-code-alpaca merging · MBPP | pass@1: 0.2240 -> 0.2239 | Hurt |
| use element-wise rather than layer-wise merging coefficients (ablation) | ProDistill · ViT-B-32 vision tasks (8-task Avg), 64 validation shots | accuracy: 72.96 -> 86.04 | Helped |
| use element-wise rather than layer-wise merging coefficients (ablation) | ProDistill · RoBERTa GLUE tasks (Avg), 64 validation shots | average score: 0.5560 -> 0.7641 | Helped |
| use dual activations instead of merged activations only (ablation) | ProDistill · ViT-B-32 vision tasks (8-task Avg) | accuracy: 85.20~(-1.0\%) -> 86.04 | Helped |
| use dual activations instead of fine-tuned activations only (ablation) | ProDistill · ViT-B-32 vision tasks (8-task Avg) | accuracy: 84.98~(-1.2\%) -> 86.04 | Helped |
| use dual activations instead of merged activations only (ablation) | ProDistill · RoBERTa GLUE tasks (Avg) | average score: 0.7303~(-4.4\%) -> 0.7641 | Helped |
| use dual activations instead of fine-tuned activations only (ablation) | ProDistill · RoBERTa GLUE tasks (Avg) | average score: 0.5895~(-22.8\%) -> 0.7641 | Helped |
Surprising findings
- Layer-wise progressive training outperforms its end-to-end counterpart DistillMerge, contrary to the common belief that layer-wise training hurts performance.
- Element-wise merging coefficients help ProDistill but hurt AdaMerging, which performs better with layer-wise coefficients.
Limitations
- Theoretical hardness results are worst-case and do not directly describe practical merging performance.
- Requires few-shot unlabeled validation data per task, unlike data-free methods such as task arithmetic.
- Evaluation of LLM merging is limited to pairwise merging of 13B models due to scale constraints; baselines for LLMs are fewer.
Experimental setup
Datasets: MNIST EuroSAT GTSRB SVHN DTD RESISC45 Stanford Cars SUN397 GLUE (CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE) AlpacaEval 2.0 GSM8K MATH
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE AlpacaEval 2.0 GSM8K MATH
Metrics: accuracy Matthews correlation pass@1 win rate normalized average
Baselines: Task Arithmetic Fisher merging RegMean AdaMerging Localize-and-Stich TIES-Merging WIDEN DistillMerge
Builds on: task arithmetic RegMean AdaMerging Localize-and-Stitch
Closest papers by meaning
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.88)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.88)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.88)
- Rethinking Layer-wise Model Merging through Chain of Merges (NeurIPS 2026 · similarity 0.88)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.