Research map / Large language models / Instruction and fine-tuning
Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Twin-Merging, which splits task knowledge into a shared expert plus SVD-compressed exclusive task vectors and dynamically merges them with a learned input-conditioned router, narrowing the gap to fine-tuned experts on GLUE, generative LLM tasks and vision tasks.
Problem
Model merging of multiple task-specific fine-tuned models into one multitask model leaves a large performance gap to individual experts, due to parameter/task interference and a static, one-size-fits-all merged model that cannot adapt to heterogeneous test inputs.
Approach
Decompose experts into a shared expert (merged base) and low-rank SVD-compressed exclusive task vectors (expert minus shared), then use a small router conditioned on the shared expert's last-layer token embeddings to softmax-weight and dynamically compose these exclusive vectors at inference.
Key results
- On 8 GLUE discriminative tasks with RoBERTa-base, Twin-Merging reaches 96.14 average normalized score vs 67.80 for Task Arithmetic.
- On 4 generative tasks with Qwen-14B LoRA experts, Twin-Merging scores 102.38 vs 96.61 for Task Arithmetic and exceeds the fine-tuned upper bound (100).
- On 8 CV tasks with ViT-B/32, Twin-Merging gets 95.33 normalized score vs 94.04 for Surgery, with 47m22s extra time and 5.0GB VRAM vs 215m01s and 32.4GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic merging of task-specific knowledge to the shared model (ablation) | Twin-Merging · GLUE (8 discriminative tasks) | average normalized score: 67.80 -> 96.14 | Helped |
| add dynamic merging of task-specific knowledge to the shared model (ablation) | Twin-Merging · Qwen generative tasks | average normalized score: 96.61 -> 102.38 | Helped |
| use a shared expert instead of a task-specific expert as the foundation for dynamic merging (ablation) | Twin-Merging · RoBERTa (8 discriminative tasks) | average normalized score: 81.47 -> 96.14 | Helped |
| use a shared expert instead of a task-specific expert as the foundation for dynamic merging (ablation) | Twin-Merging · Qwen generative tasks | average normalized score: 87.77 -> 102.38 | Helped |
| use a shared expert rather than the pretrained model as the foundation for dynamic merging (ablation) | Twin-Merging · RoBERTa (8 discriminative tasks) | average normalized score: 85.90 -> 96.14 | Helped |
| use a shared expert rather than the pretrained model as the foundation for dynamic merging (ablation) | Twin-Merging · Qwen generative tasks | average normalized score: 95.03 -> 102.38 | Helped |
| increase exclusive-vector compression rank from 1 to 16 (ablation) | Twin-Merging · Qwen generative tasks | average normalized score: 100.96 -> 102.38 | Helped |
| use SVD rather than Magnitude or Bernoulli Dropout for sparsification (ablation) | Twin-Merging · GLUE | performance | Helped |
| merge with Twin-Merging rather than Task Arithmetic | RoBERTa-base · GLUE (8 discriminative tasks) | average normalized score: 67.80 -> 96.14 | Helped |
| merge with Twin-Merging rather than Task Arithmetic (w/ DARE) | Qwen-14B · Generative tasks | average normalized score: 98.52 -> 102.38 | Helped |
| merge with Twin-Merging rather than Surgery | ViT-B/32 · CV tasks (8 image classification datasets) | average normalized score: 94.04 -> 95.33 | Helped |
| merge with Twin-Merging rather than Task Arithmetic | Qwen-72B · TruthfulQA | score: 98.70 -> 99.30 | Helped |
Surprising findings
- Twin-Merging exceeds the fine-tuned upper bound on generative tasks (102.38 vs 100), attributed to the general knowledge in Qwen-14B.
- Retaining only about 0.2% of parameters (99.8% compression) still yields 86.4% of performance on GLUE.
Limitations
- Requires a small validation set per task to train the router and relies on task-specific experts being available.
- Inference is slightly slower than conventional merging due to dynamic composition of twin vectors.
Experimental setup
Datasets: GLUE (CoLA, SST-2, MRPC, STS-B, QQP, QNLI, MNLI, RTE) MMLU TruthfulQA BBQ CNN-DailyMail Dolly-15k SUN397 Stanford Cars RESISC45 EuroSAT SVHN GTSRB
Benchmarks: GLUE MMLU TruthfulQA BBQ CNN-DailyMail SUN397 Stanford Cars RESISC45 EuroSAT SVHN GTSRB MNIST
Metrics: normalized score exact match ROUGE-2 accuracy inference time VRAM
Baselines: Weight Averaging Task Arithmetic Ties-Merging DARE AdaMerging Surgery Multi-Task Learning
Builds on: Task Arithmetic Ties-Merging DARE LoRA AdaMerging
Closest papers by meaning
- CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging (ICML 2025 · similarity 0.91)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.91)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.91)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.91)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.91)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.