Research map / Label-efficient and robust learning / Recognition and long-tail
CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CALM, a model merging method that optimizes localized binary masks over task vectors using class-balanced entropy-minimization pseudo-labeled unlabeled data and sequential merging of a few tasks, reaching near multi-task-learning accuracy on CLIP ViT-B/32 vision benchmarks.
Problem
Model merging of independently fine-tuned models into one multi-task model suffers from parameter interference with global methods and from loss of task-specific effective information with localized methods; the paper asks how to extract effective information that stays effective after merging.
Approach
Extract localized parameters via learned binary masks optimized on pseudo-labeled, class-balanced low-entropy unlabeled samples from all visible tasks, and merge sequentially with conflict-free mask-based combination ((1-M) ⊙ τ_seq + M ⊙ τ_j), with most tasks pre-merged by task arithmetic.
Key results
- On 8 vision tasks with CLIP ViT-B/32, CALM reaches 87.7% average accuracy versus 79.9% for Localize-and-Stitch and 80.1% for LW AdaMerging (traditional MTL 88.9%).
- With CLIP ViT-L/14, CALM achieves 92.1% average accuracy, slightly above Pareto Merging at 92.0% and close to traditional MTL at 93.5%.
- Removing 5% of interfering parameters from the pre-merged task vector raises average accuracy from 46.9% to 71.2% in the single-sequential-task analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CALM rather than Pareto Merging | CLIP ViT-B/32 · 8 vision classification tasks | Avg Acc: 84.8 -> 87.7 | Helped |
| use CALM rather than Pareto Merging | CLIP ViT-L/14 · 8 vision classification tasks | Avg Acc: 92.0 -> 92.1 | Helped |
| remove M* masked interference parameters from τ1 (ablation) | CALM · 8 vision classification tasks, one sequential task | Avg Acc: 46.9 -> 71.2 | Helped |
| restrict τ2 to M* masked parameters (ablation) | CALM · 8 vision classification tasks, one sequential task | Avg Acc: 90.5 -> 75.3 | Hurt |
| use only M rather than both mask parts (ablation) | CALM · 8 vision classification tasks | Avg Acc: 87.7 -> 86.6 | Hurt |
| use only 1−M rather than both mask parts (ablation) | CALM · 8 vision classification tasks | Avg Acc: 87.7 -> 85.7 | Hurt |
| set learning rate to 5e6 instead of 1e7 (ablation) | CALM · 8 vision classification tasks | Avg Acc: 87.7 -> 87.5 | No clear effect |
| set learning rate to 2e7 instead of 1e7 (ablation) | CALM · 8 vision classification tasks | Avg Acc: 87.7 -> 87.6 | No clear effect |
| set L1 regularization coefficient to 0.5 instead of 1e7 (ablation) | CALM · 8 vision classification tasks | Avg Acc: 87.7 -> 87.8 | No clear effect |
| set L1 regularization coefficient to 2 instead of 1e7 (ablation) | CALM · 8 vision classification tasks | Avg Acc: 87.7 -> 87.6 | No clear effect |
| use class-balanced entropy minimization sampling instead of unsupervised entropy minimization (ablation) | CALM · visual and NLP tasks | average accuracy | Helped |
| increase the CB-EMS sampling proportion too far (ablation) | CALM · visual and NLP tasks | average accuracy | Hurt |
Limitations
- Gains over the best baselines are modest (about 1-3%) on most tasks, and CALM slightly underperforms on DTD for ViT-L/14.
- Sequential merging order and the number of sequential tasks introduce some randomness, though the reported standard deviation is small (about 0.4%).
- Relies on unlabeled task-specific data for mask optimization, and the entropy-based pseudo-labels depend on the quality of the fine-tuned model's initial predictions.
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD SST-2 CR MR MPQA
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE subsets (SST-2, CR, MR, MPQA, TREC, SUBJ, QNLI, SNLI, MNLI, RTE, MRPC, QQP)
Metrics: average accuracy accuracy relative to individual models
Baselines: Weight Averaging RegMean Fisher Merging Task Arithmetic TW AdaMerging LW AdaMerging Pareto Merging Ties-Merging Consensus TA Consensus TIES Localize-and-Stitch
Builds on: task arithmetic Ties-Merging AdaMerging Localize-and-Stitch
Closest papers by meaning
- Localizing Task Information for Improved Model Merging and Compression (ICML 2024 · similarity 0.90)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.89)
- CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging (ICML 2025 · similarity 0.89)
- Contribution-Aware Structured Sparsity for Model Merging (NeurIPS 2026 · similarity 0.89)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.89)
- CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.