Research map / Architectures and efficiency / Pruning and efficient tuning
Localizing Task Information for Improved Model Merging and Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that task-specific information survives merging via binary task masks (TALL-masks) that localize each task vector in the multi-task vector, and proposes Consensus Merging, which removes catastrophic and selfish weights to improve task arithmetic and TIES, plus a mask-based compression scheme.
Problem
Model merging via task arithmetic loses significant performance versus individual fine-tuned models; the paper asks whether this is due to information erasure or task interference, and how to compress many fine-tuned checkpoints.
Approach
Task-specific binary masks, built by thresholding each task vector against the residual of the multi-task vector, recover nearly full single-task performance from the merged vector; mask agreement statistics reveal catastrophic (used by no task) and selfish (used by one task) weights, and removing them (consensus mask with threshold k) improves existing merging methods.
Key results
- Consensus TA improves Task Arithmetic by 4.9% absolute average accuracy on 20-task ViT-L/14 vision benchmark, and Consensus TIES improves TIES by 1.1% there.
- TALL Mask + TIES retains about 99.7% normalized accuracy... on 20-task ViT-B/32 the compression reduces storage from 57Gb to 8.2Gb while retaining 99.7% of original performance (TALL Mask + TIES reaches 91.1% absolute, 99.7% normalized).
- On 8-QA NLP benchmark, Consensus TIES improves TIES by 6.3% absolute accuracy (69.1 vs 62.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply consensus masking to Task arithmetic | Task arithmetic on 8 QA tasks · 8 QA tasks | accuracy: 63.8 -> 68.6 | Helped |
| Apply consensus masking to TIES | TIES on 8 QA tasks · 8 QA tasks | accuracy: 62.8 -> 69.1 | Helped |
| Apply consensus masking to Task arithmetic | Task arithmetic on all 11 NLP tasks · All 11 tasks | accuracy: 63.6 -> 67.5 | Helped |
| Apply consensus masking to Task arithmetic | Task arithmetic on 20 vision tasks · 20 tasks, CLIP ViT-B/32 | accuracy: 60.6 -> 65.4 | Helped |
| Apply consensus masking to Task arithmetic | Task arithmetic on 20 vision tasks · 20 tasks, CLIP ViT-L/14 | accuracy: 74.0 -> 78.9 | Helped |
| Localize task-specific parameters with TALL masks | Task arithmetic compression · 20 tasks, CLIP ViT-B/32 | accuracy: 60.6 -> 90.6 | Helped |
| Use consensus threshold k=2 rather than k=1 (ablation) | Consensus Task Arithmetic · 20 tasks, CLIP ViT-B/32 | accuracy: 64.5 -> 65.4 | Helped |
| Use consensus threshold k=2 rather than k=1 (ablation) | Consensus TIES · 20 tasks, CLIP ViT-B/32 | accuracy: 64.5 -> 63.2 | Hurt |
| Remove selfish weights as well as catastrophic weights (ablation) | Consensus merging with (IA)³ NLP models · 7 NLP tasks | accuracy: 70.9 -> 66.6 | Hurt |
| Preserve task-specific information with TALL masks rather than magnitude masking | Checkpoint compression · 20 tasks, CLIP ViT-B/32 | accuracy: 75.3 -> 90.6 | Helped |
Surprising findings
- Task arithmetic on a controlled non-overlapping fine-tuning setup (no weight interference, no information erasure) still degrades, suggesting task interference rather than information erasure is the cause.
- Optimal pruning threshold differs by method: k=2 for Consensus TA but k=1 (removing only catastrophic weights) for Consensus TIES.
Limitations
- Storing binary masks adds storage cost relative to plain task arithmetic.
- Optimal threshold k depends on task count, task combination, and merging method, so it must be tuned.
- Mask construction requires fine-tuned checkpoints and per-task validation to tune lambda.
Experimental setup
Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN CIFAR100 STL10 Flowers102 OxfordIIITPet
Benchmarks: 8-task vision benchmark 14-task vision benchmark 20-task vision benchmark 7 NLP tasks 8 QA tasks 11 NLP tasks
Metrics: absolute accuracy normalized accuracy storage (Gb)
Baselines: Weight averaging Task arithmetic TIES Magnitude Pruning Magnitude Masking Fisher merging
Builds on: task arithmetic TIES-merging magnitude pruning
Closest papers by meaning
- CALM: Consensus-Aware Localized Merging for Multi-Task Learning (ICML 2025 · similarity 0.90)
- Contribution-Aware Structured Sparsity for Model Merging (NeurIPS 2026 · similarity 0.89)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.88)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.88)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
- CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.