Research map / Label-efficient and robust learning / Recognition and long-tail
Representation Surgery in Model Merging with Probabilistic Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProbSurgery, a probabilistic (Gaussian, sampled) post-merging representation-surgery module that corrects representation bias in merged models, with a one-to-all variant using a single module across tasks.
Problem
Merged models suffer representation bias (distribution discrepancy between merged and individual models' representations), which degrades multitask performance; the deterministic Surgery module is limited in modeling capacity and is task-specific.
Approach
Model the representation bias as a Gaussian distribution whose parameters are produced by an MLP per sample, sample it via reparameterization to correct merged features, and extend to a single shared module via direct feature alignment or proxy distribution alignment with scoring rules.
Key results
- ProbSurgery applied to Weight Averaging raises ViT-B/32 eight-task average accuracy from 80.0% (Surgery) to 86.7%.
- With Task Arithmetic, ProbSurgery reaches 87.0% average vs 80.9% for Surgery on eight vision tasks.
- A single one-to-all ProbSurgery module (PDA) reaches 82.4% average, exceeding eight task-specific Surgery modules (80.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ProbSurgery instead of Surgery | Weight Averaging · 8 vision tasks (Avg.) | accuracy: 80.0 -> 86.7 | Helped |
| add ProbSurgery instead of Surgery | Task Arithmetic · 8 vision tasks (Avg.) | accuracy: 80.9 -> 87.0 | Helped |
| add ProbSurgery instead of Surgery | Ties-Merging · 8 vision tasks (Avg.) | accuracy: 83.1 -> 88.0 | Helped |
| add ProbSurgery instead of Surgery | LW AdaMerging · 8 vision tasks (Avg.) | accuracy: 87.5 -> 88.8 | Helped |
| add ProbSurgery instead of Surgery | Weight Averaging · 5 NLP tasks (Avg.) | accuracy: 74.3 -> 76.0 | Helped |
| add ProbSurgery instead of Surgery | Task Arithmetic · 5 NLP tasks (Avg.) | accuracy: 74.4 -> 75.7 | Helped |
| use one ProbSurgery module with DFA rather than eight task-specific Surgery modules (ablation) | Task Arithmetic · 8 vision tasks (Avg.) | accuracy: 80.9 -> 81.8 | Helped |
| use one ProbSurgery module with PDA rather than eight task-specific Surgery modules (ablation) | Task Arithmetic · 8 vision tasks (Avg.) | accuracy: 80.9 -> 82.4 | Helped |
| increase hidden-layer width from 128 to 2048 (ablation) | ProbSurgery · 8 vision tasks | accuracy: 86.43 -> 85.94; 87.14 -> 87.65; 87.74 -> 88.52; 88.47 -> 88.52 | Hurt |
Surprising findings
- A single shared ProbSurgery module outperforms eight independent task-specific Surgery modules.
Limitations
- Gains are reported primarily on vision tasks with CLIP ViT backbones and a small set of NLP tasks.
- Theoretical bound is a PAC-Bayes argument that compares to a deterministic baseline with loose assumptions.
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD AG News Yelp Sentiment Amazon Sentiment Yahoo Q&A
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD AG News Yelp Sentiment Amazon Sentiment Yahoo Q&A
Metrics: test accuracy average accuracy L1 distance AUROC
Baselines: Weight Averaging Fisher Merging RegMean Task Arithmetic Ties-Merging Concrete TA Concrete AM AdaMerging Surgery EMR-MERGING KD4MTL
Builds on: Surgery Task Arithmetic Ties-Merging AdaMerging PAC-Bayes
Closest papers by meaning
- Representation Surgery for Multi-Task Model Merging (ICML 2024 · similarity 0.89)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.85)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.85)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.85)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.84)
- Parameter Competition Balancing for Model Merging (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.