Research map / Label-efficient and robust learning / Recognition and long-tail
Representation Surgery for Multi-Task Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies "representation bias" between merged and individual models in model-merging-based multi-task learning and proposes Surgery, a lightweight task-specific adapter trained with an unsupervised objective on unlabeled test data to reduce it, improving multiple merging methods on ViT and BERT benchmarks.
Problem
Model merging for multi-task learning (merging independently fine-tuned models without raw data) lags far behind traditional MTL and individual models, and the merged model's representations deviate from those of the individual models ("representation bias").
Approach
A task-specific Adapter-like module placed after the merged backbone takes the merged representation and subtracts estimated task-specific bias, trained without labels by minimizing L1 distance between the corrected merged representation and the frozen individual model's representation on unlabeled test inputs; it is orthogonal to and plug-in for any weight-space merging method.
Key results
- On ViT-B/32 eight-task merging, AdaMerging rises from 80.1% to 86.1% average accuracy with Surgery (87.5% with rank 64), close to Traditional MTL's 88.9%.
- Task Arithmetic improves from 69.1% to 80.9% average accuracy on ViT-B/32 with Surgery, and reduces EuroSAT L1 representation bias from 0.30 to 0.13.
- On BERT five-task NLP merging, Task Arithmetic improves from 64.3% to 74.4% average accuracy, close to Traditional MTL at 75.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Surgery to Task Arithmetic | ViT-B/32 merged model · Eight-task average | accuracy: 69.1 -> 80.9 | Helped |
| Add Surgery to AdaMerging | ViT-B/32 merged model · Eight-task average | accuracy: 80.1 -> 87.5 | Helped |
| Add Surgery to AdaMerging | ViT-L/14 merged model · Eight-task average | accuracy: 90.8 -> 92.3 | Helped |
| Add Surgery to Weight Averaging | BERT merged model · Five NLP tasks average | accuracy: 55.8 -> 74.3 | Helped |
| Add Surgery to Task Arithmetic | BERT merged model · Five NLP tasks average | accuracy: 64.3 -> 74.4 | Helped |
| Increase Surgery rank from 16 to 64 (ablation) | AdaMerging, ViT-B/32 · Eight-task average | accuracy: 86.1% to 87.5% | Helped |
| Increase Surgery rank from 4 to 64 (ablation) | Task Arithmetic, ViT-B/32 · Eight-task average | accuracy: 74.3% to 84.3% | Helped |
| Use MSELoss instead of L1 Loss in Surgery (ablation) | Task Arithmetic, ViT-B/32 · Eight-task average | accuracy: 80.9 -> 81.4 | Helped |
| Use SmoothL1Loss instead of L1 Loss in Surgery (ablation) | Task Arithmetic, ViT-B/32 · Eight-task average | accuracy: 80.9 -> 81.6 | Helped |
| Use Negative CosineSimilarity instead of L1 Loss in Surgery (ablation) | Task Arithmetic, ViT-B/32 · Eight-task average | accuracy: 80.9 -> 81.2 | Helped |
| Increase offline available test data from 1% to 100% (ablation) | AdaMerging w/ Surgery · Eight-task average, ViT-B/32 | accuracy: 82.8 -> 86.1 | Helped |
| Increase online available test data from 10% to 50% (ablation) | AdaMerging w/ Surgery · Eight-task average, ViT-B/32 | accuracy: 83.3% to 84.8% | Helped |
Surprising findings
- Representation bias (L1 distance) decreases consistently as merging methods improve, from Weight Averaging to Task Arithmetic to AdaMerging, suggesting it is a key obstacle to merged MTL performance.
- Surgery adds only about 0.01-0.014% parameters relative to the merged model, yet recovers most of the gap to traditional MTL.
Limitations
- Requires unlabeled test data from all tasks at merge time, which weakens the privacy motivation of model merging.
- Surgery module is task-specific, so it adds per-task parameters and must be stored and applied per task at inference.
- Evaluated only on ViT CLIP vision models with eight tasks and BERT with five NLP tasks; the authors note future work on other architectures or initializations.
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD AG News Yelp Sentiment Amazon Sentiment Yahoo Q&A
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD AG News Yelp Sentiment Amazon Sentiment Yahoo Q&A
Metrics: accuracy average accuracy L1 representation bias
Baselines: Weight Averaging Fisher Merging RegMean Task Arithmetic Ties-Merging Concrete TA Concrete AM TW AdaMerging AdaMerging Traditional MTL KD4MTL
Builds on: Task Arithmetic Ties-Merging AdaMerging Adapter KD4MTL
Closest papers by meaning
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.90)
- Representation Surgery in Model Merging with Probabilistic Modeling (ICML 2025 · similarity 0.89)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.89)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.89)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.