Research map / Architectures and efficiency / Pruning and efficient tuning
Task Singular Vectors: Reducing Task Interference in Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Task Singular Vectors: layer-wise SVD of task matrices, with TSV-Compress (10% storage, 99% accuracy) and TSV-Merge (whitening/Procrustes-based interference reduction on truncated singular vectors) for training-free model merging.
Problem
Task Arithmetic-style model merging treats networks as flat parameter vectors, ignoring layer structure and suffering from task interference; cosine similarity is too coarse for inter-task interaction.
Approach
Analyze per-layer task matrices via SVD, keep only top 1/T singular components per task, define Singular Task Interference from the overlap of concatenated singular vectors, and decorrelate them with an orthogonal Procrustes (whitening) step before reconstructing the merged layer.
Key results
- TSV-Merge on ViT-L-14 (8 tasks) reaches 92.98% absolute accuracy and 96.98% normalized accuracy, versus 84.93% for Task Arithmetic.
- TSV-Compress keeps over 99% of fine-tuned accuracy while storing about 10% of task vector size (roughly twice the original model size in total).
- On ViT-B-32, TSV-M outperforms Task Arithmetic and Consensus TA by an average of +15.45% and +10.71% absolute accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Consensus TA with TSV-Merge | ViT-B-32 model merging · 8, 14, and 20 tasks | average absolute accuracy: 75.03 to 85.86; 70.39 to 80.06; 65.43 to 77.07 | Helped |
| replace Consensus TA with TSV-Merge | ViT-B-16 model merging · 8, 14, and 20 tasks | average absolute accuracy: 79.39 to 89.01; 74.39 to 84.58; 69.76 to 80.57 | Helped |
| replace Consensus TA with TSV-Merge | ViT-L-14 model merging · 8, 14, and 20 tasks | average absolute accuracy: 86.34 to 92.98; 82.22 to 89.17; 79.00 to 87.72 | Helped |
| add low-rank approximation without interference reduction (ablation) | Task Arithmetic on ViT-B-32 · 8 tasks | average absolute accuracy: 76.5 to 75.2 (-1.3) | Hurt |
| add low-rank approximation without interference reduction (ablation) | Task Arithmetic on ViT-B-32 · 14 tasks | average absolute accuracy: 72.1 to 71.0 (-1.1) | Hurt |
| add low-rank approximation without interference reduction (ablation) | Task Arithmetic on ViT-B-32 · 20 tasks | average absolute accuracy: 66.8 to 66.3 (-0.5) | Hurt |
| add interference reduction without low-rank approximation (ablation) | Task Arithmetic on ViT-B-32 · 8 tasks | average absolute accuracy: 76.5 to 82.6 (+7.4) | Helped |
| add interference reduction without low-rank approximation (ablation) | Task Arithmetic on ViT-B-32 · 14 tasks | average absolute accuracy: 72.1 to 75.7 (+4.7) | Helped |
| add interference reduction without low-rank approximation (ablation) | Task Arithmetic on ViT-B-32 · 20 tasks | average absolute accuracy: 66.8 to 69.9 (+3.6) | Helped |
| add both low-rank approximation and interference reduction (ablation) | Task Arithmetic on ViT-B-32 · 8 tasks | average absolute accuracy: 76.5 to 92.3 (+9.7) | Helped |
| add both low-rank approximation and interference reduction (ablation) | Task Arithmetic on ViT-B-32 · 14 tasks | average absolute accuracy: 72.1 to 87.9 (+12.2) | Helped |
| add both low-rank approximation and interference reduction (ablation) | Task Arithmetic on ViT-B-32 · 20 tasks | average absolute accuracy: 66.8 to 84.3 (+14.4) | Helped |
Surprising findings
- Orthogonalizing full-rank task singular vectors via Procrustes incurs large approximation error, so interference reduction works better on low-rank truncated components.
- Merging coefficient alpha = 1.0 works best consistently, removing the need for scaling tuning.
Limitations
- Uniform rank across tasks (1/T of dimensionality) rather than per-task optimal rank.
- Evaluated only on CLIP vision models and image classification tasks.
- Compression requires known or inferred task identity, whereas TSV-M does not.
Experimental setup
Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN CIFAR100 STL10 Flowers102 OxfordIIITPet
Benchmarks: 8-task vision merging benchmark 14-task vision merging benchmark 20-task vision merging benchmark
Metrics: average absolute accuracy average normalized accuracy singular task interference
Baselines: Weight Averaging Task Arithmetic Consensus TA Zero-shot TALL-Mask+TIES Ties-Merging DARE Fisher Merging RegMean
Builds on: Task Arithmetic Consensus merging (TALL-masks) Eckart-Young theorem Ties-Merging DARE
Closest papers by meaning
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.90)
- Less is More: Efficient Model Merging with Binary Task Switch (CVPR 2025 · similarity 0.88)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.88)
- Localizing Task Information for Improved Model Merging and Compression (ICML 2024 · similarity 0.87)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.87)
- Contribution-Aware Structured Sparsity for Model Merging (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.