Research map / Architectures and efficiency / Pruning and efficient tuning
Less is More: Efficient Model Merging with Binary Task Switch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows task vectors have a pulse-like magnitude profile, binarizes them into a mask, sign and scalar ("T-Switch") and combines them training-free via KNN retrieval on a small query set ("Auto-Switch") for model merging at 1-3% of full-precision storage.
Problem
Model merging of fine-tuned task vectors suffers from parameter conflicts between tasks and high storage cost of storing full-precision task vectors for dynamic merging.
Approach
Task vectors exhibit a pulse-like property where only large-magnitude parameters matter; discarding small ones and binarizing the rest into a mask, sign matrix and scalar knob reduces conflicts and storage, and a training-free KNN-based query mechanism assigns per-task switch weights at inference.
Key results
- Auto-Switch reaches 90.39% average accuracy on 8 vision tasks with ViT-B/32 full-rank task vectors, outperforming EMR-merging (88.74%) and Twin-merging (83.07%) at 58.6 MB storage.
- T-Switch achieves 91.14% average on ViT-B/32, slightly above the individual fine-tuned models (91.01%) at a discard ratio of 0.7.
- On GLUE with RoBERTa-base, Auto-Switch reaches 0.8421 average, about 2.26 points above Twin-merging, using 65.5 MB (1.71% of Twin-merging's extra storage).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Auto-Switch instead of Twin-merging for full-rank task-vector merging | CLIP ViT-B/32 · Eight vision datasets (average) | accuracy: 83.07 -> 90.39 | Helped |
| Use Auto-Switch instead of EMR-merging for full-rank task-vector merging | CLIP ViT-B/32 · Eight vision datasets (average) | accuracy: 88.74 -> 90.39 | Helped |
| Use Auto-Switch instead of Twin-merging for low-rank task-vector merging | CLIP ViT-B/32 LoRA · Eight vision datasets (average) | accuracy: 85.86 -> 84.96 | Hurt |
| Use Auto-Switch instead of EMR-merging for low-rank task-vector merging | CLIP ViT-B/32 LoRA · Eight vision datasets (average) | accuracy: 84.46 -> 84.96 | Helped |
| Use Auto-Switch instead of Twin-merging | RoBERTa-base · Eight GLUE datasets (average) | average task score: 0.8235 -> 0.8421 | Helped |
| Discard low-magnitude parameters rather than randomly discarding parameters (ablation) | ViT-B/32 task-vector merging · Eight vision datasets (average) | accuracy: At α=0.1: 69.31 vs 69.06; at α=0.8: 70.99 vs 66.09 | Mixed |
| Increase low-magnitude parameter discard ratio with P-Discard (ablation) | ViT-B/32 task-vector merging · Eight vision datasets (average) | accuracy: 69.31 at 0.1 -> 72.23 at 0.7 | Helped |
| Increase discard ratio beyond the useful threshold (ablation) | T-Switch on CLIP ViT-B/32 · Eight vision datasets | accuracy | Hurt |
| Increase discard ratio on RoBERTa T-Switch (ablation) | RoBERTa-base · Eight GLUE datasets | average task score: highest ... 0.8446 at a discard ratio of 0.4; fine-tuning ... 0.8556 | Mixed |
| Replace full-precision task vectors with Bin-Discard binary approximations (ablation) | Task-vector merging · Eight vision datasets | accuracy | No clear effect |
Surprising findings
- Discarding up to 60-70% of the smallest-magnitude task vector parameters and binarizing the rest can match or exceed the performance of full-precision fine-tuned models.
- Random DARE-style dropping shows no performance gain as drop rate rises, while magnitude-based dropping improves merging performance.
Limitations
- Auto-Switch slightly underperforms T-Switch on scene datasets (SUN397, RESISC45) due to KNN resolution with high inter-class similarity.
- Performance is sensitive to discard ratio; too high a ratio loses essential task information.
- Binarization evaluated mainly on CLIP vision models and RoBERTa-base GLUE tasks.
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE (CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE)
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE
Metrics: classification accuracy Matthews correlation Pearson/Spearman average storage (MB)
Baselines: Weight-Averaging Task-Arithmetic Ties-Merging DARE RegMean Fisher Merging AdaMerging AdaMerging++ Twin-merging EMR-merging
Builds on: Task Arithmetic TIES-Merging DARE EMR-Merging Twin-Merging
Closest papers by meaning
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.90)
- Parameter Competition Balancing for Model Merging (NeurIPS 2024 · similarity 0.88)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.88)
- Task Singular Vectors: Reducing Task Interference in Model Merging (CVPR 2025 · similarity 0.88)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.88)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.