Research map / Architectures and efficiency / Pruning and efficient tuning
CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAT Merging, a training-free model merging method that trims conflict-prone components of other tasks' task vectors using projection (linear weights) and masking (norm scale/shift), guided by a few unlabeled exemplars per task.
Problem
Multi-task model merging via task arithmetic suffers knowledge conflicts among task vectors, degrading performance on individual tasks; existing trimming methods remove low-magnitude components but overlook high-magnitude components that cause conflict.
Approach
Derive a layer-wise bound on knowledge conflict and select, per task, a removal basis (linear layers) or removal mask (normalization scale and shift parameters) from other task vectors that maximizes feature energy on the task's own inputs while minimizing it on other tasks' inputs, using only a few unlabeled exemplars.
Key results
- CAT Merging reaches 78.3% average accuracy on eight vision tasks with ViT-B/32 (vs 75.8% for PCB Merging) and 89.6% with ViT-L/14 (vs 87.6% for PCB Merging).
- Ablation on ViT-B/32: removing linear-weight trimming drops average accuracy to 71.5%, while the full method reaches 78.3%.
- With only one exemplar per task, CAT Merging still reaches 76.61% (ViT-B/32) and 89.01% (ViT-L/14).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CAT Merging instead of PCB Merging | ViT-B/32 model merging · Eight vision tasks, average accuracy | accuracy: 75.8 -> 78.3 | Helped |
| Apply CAT Merging instead of PCB Merging | ViT-L/14 model merging · Eight vision tasks, average accuracy | accuracy: 87.6 -> 89.6 | Helped |
| Apply CAT Merging instead of PCB Merging | RoBERTa model merging · GLUE tasks, Average | score: 59.59 -> 62.56 | Helped |
| Apply CAT Merging instead of Task Arithmetic | ViT-B/16 model merging · Eight vision tasks, average performance | accuracy: 73.8 -> 82.1 | Helped |
| Add linear weight trimming (ablation) | CAT Merging · Eight vision tasks, average accuracy (ViT-B/32) | accuracy: 71.5 -> 78.3 | Helped |
| Add linear weight trimming (ablation) | CAT Merging · Eight vision tasks, average accuracy (ViT-L/14) | accuracy: 85.6 -> 89.6 | Helped |
| Add normalization scaler trimming (ablation) | CAT Merging · Eight vision tasks, average accuracy (ViT-B/32) | accuracy: 76.8 -> 78.3 | Helped |
| Add normalization scaler trimming (ablation) | CAT Merging · Eight vision tasks, average accuracy (ViT-L/14) | accuracy: 89.2 -> 89.6 | Helped |
| Add shift trimming (ablation) | CAT Merging · Eight vision tasks, average accuracy (ViT-B/32) | accuracy: 77.8 -> 78.3 | Helped |
| Add shift trimming (ablation) | CAT Merging · Eight vision tasks, average accuracy (ViT-L/14) | accuracy: 88.7 -> 89.6 | Helped |
Surprising findings
- Trimming high-magnitude components, not just low-magnitude ones, is needed; magnitude-based pruning overlooks conflicts from large components.
- CAT Merging does not win on every dataset; Fisher Merging and PCB Merging do better on some tasks such as Cars and SVHN/MNIST.
Limitations
- Requires hyperparameters lambda and c (and alpha) that are tuned; performance sensitive to alpha.
- Does not outperform all baselines on every individual task (e.g., Fisher Merging better on Cars, PCB on SVHN and MNIST).
- Vision-language and NLP results are mixed (e.g., lower on MRPC and STS-B for RoBERTa, lower TextCaps for BLIP).
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD COCO Caption Flickr30k Caption TextCaps OKVQA
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD COCO Caption Flickr30k Caption TextCaps OKVQA
Metrics: accuracy CIDEr average accuracy standard deviation of accuracy drops
Baselines: Weight Averaging Fisher Merging RegMean Task Arithmetic Ties-Merging TATR Ties-Merging & TATR Consensus Merging PCB Merging AdaMerging Surgery Merging
Builds on: Task Arithmetic Ties-Merging PCB Merging Fisher Merging RegMean
Closest papers by meaning
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.91)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.91)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.91)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.90)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.89)
- CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.