Research map / Vision-language and multimodal / Multimodal LLMs
EMR-Merging: Tuning-Free High-Performance Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMR-Merging, a tuning-free model merging method that elects a unified task vector and adds lightweight per-task binary masks and scalar rescalers to approximate each task-specific model, evaluated across vision, NLP, PEFT and multimodal settings.
Problem
Model merging of multiple fine-tuned models into one usually suffers large performance degradation relative to individual models, and the best existing methods need extra data or training for tuning.
Approach
Instead of forcing one merged model to represent all tasks, elect a unified task vector (sign by total magnitude, max absolute value per parameter) and store per-task 1-bit masks (sign alignment) and scalar rescalers (magnitude alignment) that modulate it to approximate each task-specific vector, with no data or training.
Key results
- On merging 8 ViT-B/32 models, EMR-Merging reaches 88.7% average accuracy vs 81.1% for AdaMerging++ and 73.6% for Ties-Merging, close to individual models (90.5%) and traditional MTL (88.9%) without any data or training.
- On merging 30 ViT-B/16 models, EMR-Merging averages 89.54% vs 68.14% for RegMean (best baseline) and 93.02% for individual models, reducing the gap to 3.48%.
- On merging (IA)^3 PEFT modules on eleven NLP tasks, EMR-Merging achieves 67.1% average accuracy, outperforming Ties-Merging (66.4%, with validation) without validation data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMR-Merging instead of AdaMerging++ | ViT-B/32 model merging · 8 vision tasks | Avg Acc: 81.1 -> 88.7 | Helped |
| Use EMR-Merging instead of AdaMerging++ | ViT-L/14 model merging · 8 vision tasks | Avg Acc: 91.0 -> 93.7 | Helped |
| Use EMR-Merging instead of RegMean | ViT-B/16 model merging · 30 vision tasks | Average Acc: 68.14 -> 89.54 | Helped |
| Use EMR-Merging instead of Task Arithmetic | GPT-2 model merging · GLUE, 7 tasks | Avg.: 70.0 -> 80.4 | Helped |
| Use EMR-Merging instead of Ties-Merging without validation | (IA)^3 model merging · 11 NLP tasks | Avg Acc: 64.9 -> 67.1 | Helped |
| Add masking to the elected unified task vector (ablation) | EMR-Merging · 8 vision tasks, ViT-B/32 | Avg Acc: 59.3 -> 86.8; [$\uparrow$ 27.5] | Helped |
| Add rescaling to the elected unified task vector (ablation) | EMR-Merging · 8 vision tasks, ViT-B/32 | Avg Acc: 59.3 -> 71.9; [$\uparrow$ 12.6] | Helped |
| Apply masking and rescaling together (ablation) | EMR-Merging · 8 vision tasks, ViT-B/32 | Avg Acc: 59.3 -> 88.7; [$\uparrow$ 29.4] | Helped |
| Add masking and rescaling after Task Arithmetic (ablation) | Task Arithmetic · 8 vision tasks, ViT-B/32 | Avg Acc: 70.1 -> 76.9; [$\uparrow$ 6.8] | Helped |
| Add masking and rescaling after Ties-Merging (ablation) | Ties-Merging · 8 vision tasks, ViT-B/32 | Avg Acc: 73.6 -> 78.6; [$\uparrow$ 5.0] | Helped |
| Add masking and rescaling after AdaMerging++ (ablation) | AdaMerging++ · 8 vision tasks, ViT-B/32 | Avg Acc: 81.1 -> 87.7; [$\uparrow$ 6.6] | Helped |
| Apply DARE random dropping when merging eight RoBERTa models (ablation) | RegMean and Task Arithmetic · GLUE, 8 tasks | task performance | Hurt |
Surprising findings
- DARE generally fails when merging eight models, often degrading performance even though it helps in two- or three-model merges.
- EMR-Merging outperforms individual fine-tuned models in the 2-task ViT-B/32 setting (77.5 vs 76.5 average).
Limitations
- Requires extra storage for per-task masks and rescalers, though small.
- Applies only to task-vector-based settings from a shared pretrained model; cannot generalize to models trained from scratch or different architectures.
- Performance gap to individual models remains on some tasks (e.g., Cars and DTD in the 8-ViT setting).
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE ImageNet-21k COCO VQAv2
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD 30-task ViT benchmark GLUE (8 tasks) FusionBench GPT-2 seven GLUE tasks T0-3B (IA)^3 eleven NLP tasks
Metrics: accuracy Matthews correlation BLEU4 CIDEr METEOR ROUGE-L sign conflicts L2 distance cosine similarity
Baselines: Weight Averaging Fisher Merging RegMean Task Arithmetic Ties-Merging AdaMerging AdaMerging++ DARE Traditional MTL
Builds on: Task Arithmetic Ties-Merging AdaMerging DARE RegMean
Closest papers by meaning
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.90)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.90)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.89)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.89)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.89)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.