Research map / Architectures and efficiency / RNNs and normalization
Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WUDI-Merging, a data-free model merging method that minimizes interference by treating task vectors as an approximate linear subspace of layer inputs, solved by Adam over merged task vector per linear layer.
Problem
Parameter interference between task-specific expert models degrades merged model performance; existing data-free merging methods lag behind test-time adaptation methods, which need unlabeled test data.
Approach
Theoretically argues that a linear layer's task vector approximates a linear subspace of its inputs, so interference can be minimized using only task vectors by optimizing the merged task vector to reduce (tau_m - tau_i) tau_i^T errors, weighted per task by inverse squared Frobenius norm, with no data, rescaling, or regularization coefficient.
Key results
- On 8-task ViT-B/32 vision merging, WUDI-Merging reaches 85.2% average accuracy vs 76.3% for PCB Merging (data-free SOTA) and 81.1% for AdaMerging++.
- On RoBERTa-Base GLUE merging, WUDI-Merging scores 85.3 normalized vs 64.7 for Ties-Merging, a 19.7-point gain.
- Solving ViT-B/32 with 300 Adam steps takes about 1min54s and 3.03GB GPU memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WUDI-Merging instead of PCB Merging | ViT-B/32 8-task vision merging · 8-task vision benchmark | Avg Acc: 76.3 -> 85.2; 8.9% performance improvement | Helped |
| use WUDI-Merging instead of AdaMerging++ | ViT-B/32 8-task vision merging · 8-task vision benchmark | Avg Acc: 81.1 -> 85.2; 4.1% | Helped |
| use WUDI-Merging instead of Ties-Merging (w/ DARE) | RoBERTa-Base · 8-task GLUE benchmark | average normalized score: 65.6 -> 85.3; 19.7% | Helped |
| use WUDI-Merging instead of Ties-Merging (w/ DARE) | RoBERTa-Large · 8-task GLUE benchmark | average normalized score: 72.8 -> 88.8; 16.0% | Helped |
| use WUDI-Merging instead of TIES-Merging (w/ DARE) | decoder-based language model merge · AlpacaEval, GSM8K, MATH, HumanEval, MBPP | Avg.: 87.2 -> 91.5; 4.3% | Helped |
| use WUDI-Merging instead of AdaMerging++ | Flan-T5-base LoRA merge · 8-task GLUE benchmark | Avg.: 78.3 -> 81.2; 2.99% | Helped |
| use WUDI-Merging instead of Ties-Merging (w/ DARE) | Qwen-14B LoRA merge · MMLU, TruthfulQA, BBQ, CNN | Avg.: 54.62 -> 55.69; 0.99% | Helped |
| add balanced weighting to the loss (ablation) | WUDI-Merging on RoBERTa-Base · 8-task GLUE benchmark | average normalized score: 77.4 -> 85.3 | Helped |
| add balanced weighting to the loss (ablation) | WUDI-Merging on RoBERTa-Large · 8-task GLUE benchmark | average normalized score: 82.4 -> 88.8 | Helped |
| use Adam optimization instead of the closed-form solution (ablation) | WUDI-Merging on ViT-B/32 · ViT-B/32 | Accuracy: 84.4 -> 85.2 | Helped |
| vary the number of solution steps (ablation) | WUDI-Merging · vision datasets and models | accuracy | Helped |
| increase the number of merged tasks (ablation) | WUDI-Merging and merging baselines · multi-task merging subsets | average accuracy | Hurt |
Surprising findings
- Adding a balanced per-task loss weighting raises RoBERTa-Base GLUE from 77.4 to 85.3, showing task scale matters for interference.
- Code generation performance does not improve over baselines, which the authors attribute to conflicts between math and code tasks.
Limitations
- Gains are smaller on LoRA-finetuned models (Flan-T5, Qwen-14B), which the authors attribute to random LoRA initialization adding noise to task vectors.
- Code generation performance does not show an advantage, and language-task interference is attributed to input-domain overlap without a solution.
- Theoretical analysis relies on assumptions (Lipschitz continuity, bounded gradients) and an empirical coefficient replaces the reconstruction constant.
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE AlpacaEval GSM8K MATH
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE AlpacaEval GSM8K MATH
Metrics: average accuracy normalized score relative error
Baselines: Weight Averaging Fisher Merging RegMean Task Arithmetic Ties-Merging Consensus Merging PCB Merging AdaMerging AdaMerging++ Representation Surgery DARE
Builds on: Task Arithmetic RegMean AdaMerging Representation Surgery
Closest papers by meaning
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.91)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.91)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.90)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.90)
- Task Singular Vectors: Reducing Task Interference in Model Merging (CVPR 2025 · similarity 0.90)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.