Research map / Architectures and efficiency / Pruning and efficient tuning
CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CABS, a task-vector sparsification framework for model merging that combines sequential conflict-aware masking (zero overlap between task vectors) with n:m balanced pruning to outperform magnitude/random pruning and task arithmetic/TIES/DARE baselines on GLUE-based RoBERTa and LLM leaderboard Mistral-7B merges.
Problem
Magnitude-based pruning of task vectors for model merging produces high parameter overlap between task vectors and unbalanced weight distributions across layers, causing it to underperform random pruning (DARE) and leaving conflicts unresolved.
Approach
Sequentially prune task vectors, masking out parameters already retained by earlier task vectors so retained weights are non-overlapping (Conflict-Aware Sparsification), and apply n:m block-wise magnitude pruning so retained weights are evenly distributed (Balanced Sparsification).
Key results
- On RoBERTa four-task merge at 90% sparsity, CABS reaches 81.70 average vs 79.55 for Task Arithmetic (+2.15) and 79.88 for best DARE-based baseline.
- On Mistral-7B LLM Leaderboard merge at 75% sparsity, CABS averages 76.50, exceeding the ideal model (76.30) and the best baseline Task Arithmetic (76.02).
- CABS-based merged qwen2.5-7b models reportedly took the top four positions among 8B-or-fewer models on Open LLM Leaderboard as of Feb 24, 2025.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CABS instead of Task Arithmetic | RoBERTa merging · Four-task set (CoLA, SST-2, RTE, MRPC) | average accuracy: 79.55 -> 81.70 (+2.15) | Helped |
| Use CABS instead of Task Arithmetic | RoBERTa merging · Six-task set | average accuracy: 66.56 -> 69.62 (+3.06) | Helped |
| Use CABS instead of Task Arithmetic | Mistral-7B merging · LLM Leaderboard | average score: 76.02 -> 76.50 | Helped |
| Use CABS instead of Task Arithmetic | RoBERTa merging · RTE-MRPC | average accuracy: 80.15 -> 81.49 (+1.34) | Helped |
| Use block-based Balanced Sparsification instead of layer-wise magnitude pruning | RoBERTa task-vector merging · RTE-MRPC, sparsity=0.9 | average accuracy: 80.38 -> 81.30 | Helped |
| Remove Balanced Sparsification (ablation) | CABS · Sparsity ablation, 25% | average accuracy: 76.48 -> 76.29 | Hurt |
| Remove Conflict-Aware Sparsification (ablation) | CABS · Sparsity ablation, 25% | average accuracy: 76.48 -> 76.33 | Hurt |
| Remove Balanced Sparsification (ablation) | CABS · Sparsity ablation, 75% | average accuracy: 76.50 -> 76.21 | Hurt |
| Remove Conflict-Aware Sparsification (ablation) | CABS · Sparsity ablation, 75% | average accuracy: 76.50 -> 76.24 | Hurt |
| Increase task-vector overlap (ablation) | Random-pruned task vectors · RTE-MRPC and CoLA-SST2 task pairs | merged-model performance | Hurt |
| Use 16:64 or 64:256 pruning instead of 32:128 (ablation) | CABS · LLM Leaderboard, sparsity=0.75 | average score: 76.50; 76.32(+0.02); 76.39(+0.09) | Mixed |
| Change pruning order from wild-first to west-first (ablation) | CABS (32:128) · LLM Leaderboard, sparsity=0.75 | average score: 76.50(+0.20) -> 76.42(+0.12) | No clear effect |
Surprising findings
- Magnitude pruning with rescaling can recover performance comparable to DARE, contrary to the earlier finding that magnitude pruning underperforms random pruning in merging.
- Advanced pruning methods SparseGPT and Wanda perform similarly to the worst layer-wise magnitude pruning when used for task vector sparsification.
Limitations
- Gains on LLM leaderboard are small (about 0.2 points) and within the noise of the lambda grid search.
- When per-vector sparsity levels sum above 1, zero overlap is impossible and CABS only minimizes overlap.
- Improvements over the ideal-model upper bound are modest; the paper notes the bound is somewhat artificial.
Experimental setup
Datasets: GLUE (CoLA, SST-2, MRPC, RTE) RACE SQuAD AI2 ARC HellaSwag MMLU TruthfulQA Winogrande GSM8K IFEval BBH MATH
Benchmarks: GLUE (CoLA, SST-2, MRPC, RTE) RACE SQuAD Open LLM Leaderboard (ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K) Open LLM Leaderboard 2 (IFEval, BBH, MATH, GPQA, MUSR, MMLU-PRO) kobest_copa kobest_boolq
Metrics: accuracy average accuracy overlap rate
Baselines: Task Arithmetic TIES-Merging DARE Magnitude Pruning SparseGPT Wanda Ideal Model
Builds on: Task Arithmetic TIES-Merging DARE n:m pruning magnitude pruning
Closest papers by meaning
- Contribution-Aware Structured Sparsity for Model Merging (NeurIPS 2026 · similarity 0.91)
- CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging (ICML 2025 · similarity 0.89)
- CALM: Consensus-Aware Localized Merging for Multi-Task Learning (ICML 2025 · similarity 0.89)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.89)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
- Localizing Task Information for Improved Model Merging and Compression (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.