Research map / Architectures and efficiency / Pruning and efficient tuning

Contribution-Aware Structured Sparsity for Model Merging

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CASS, a contribution-aware structured mask over attention heads and FFN neurons that filters task vectors before merging (CASS-Merging) or constrains fine-tuning gradients (CASS-Tuning), improving eight merging baselines on ViT, RoBERTa and Qwen2.5.

Problem

Model merging suffers from parameter interference between conflicting task vectors; existing sparsification treats Transformers as unstructured parameter bags and ignores component-level modularity.

Approach

Score attention heads and FFN neurons by their contribution to the residual stream on small unlabeled probe sets, build structured binary masks, and zero out task-vector updates of inactive components (reverting them to pre-trained weights) before merging, or mask gradients during fine-tuning.

Key results

• CASS-Merging improves TA on ViT-B/16 from 65.02 to 69.99 average accuracy over 20 tasks (+4.97), and improves all eight baselines on both ViT backbones.
• On RoBERTa, CASS-Merging improves TA by +3.76 (66.36 to 70.12) across 8 tasks.
• CASS-Tuning improves TIES on Qwen2.5-0.5B by +0.121 normalized performance (0.695 to 0.816).

Surprising findings

• Random structured masks at the same sparsity budget do not consistently improve over TA and can hurt RoBERTa performance.
• Iso-C performance decreases slightly with CASS-Merging on Qwen2.5-0.5B (-0.009).

Limitations

• Gains on strong weight-space methods are small (e.g., +0.27 for Iso-C on ViT-B/32).
• Qwen2.5 evaluated only at 0.5B and 1.5B scale with four tasks; limitations are deferred to an appendix not included in the text.

Experimental setup

Datasets: FusionBench 20 vision tasks 8 NLU tasks (GLUE-style) MergeBench (instruction following, math, coding, safety)

Benchmarks: FusionBench 20 vision tasks RoBERTa 8-task NLU MergeBench Qwen2.5 4 tasks MMLU

Metrics: average accuracy average normalized performance OOD accuracy

Baselines: Task Arithmetic TIES-Merging DARE PCB-Merging TSV-Merging Iso-C WUDI-Merging AdaMerging

Builds on: Task Arithmetic TIES-Merging DARE TSV-Merging WUDI-Merging

Closest papers by meaning

• CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging (ICML 2025 · similarity 0.91)
• No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.89)
• Localizing Task Information for Improved Model Merging and Compression (ICML 2024 · similarity 0.89)
• Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.89)
• CALM: Consensus-Aware Localized Merging for Multi-Task Learning (ICML 2025 · similarity 0.89)
• Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.89)