Research map / Vision-language and multimodal / Human and embodied modeling
No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Iso-C and Iso-CTS, training-free model merging methods that flatten the singular value spectrum of the summed task matrices and add task-specific singular directions, improving alignment and merged accuracy on vision and NLP benchmarks.
Problem
Merging task-specific fine-tuned models into one multi-task model leaves a significant performance gap relative to single-task models; the paper seeks the properties of task matrices that enable effective merging.
Approach
Introduces the Subspace Alignment Ratio (SAR) and shows merged accuracy correlates with alignment; isotropic merging (Iso-C) replaces the singular values of the Task Arithmetic sum with their mean, and Iso-CTS combines top common singular directions with per-task orthogonal singular directions before isotropic scaling.
Key results
- Iso-CTS reaches 90.1% absolute accuracy on 20 tasks with ViT-L/14, versus 79.0% for Consensus TA and 87.7% for TSV-M.
- On 8-task LoRA merging with ViT-L/14, Iso-CTS reaches 85.3 normalized accuracy versus 78.2 for the best KnOTS variant (KnOTS-TIES).
- On T5-Large-LM-Adapt with 8 NLP tasks, Iso-C reaches 75.6 absolute accuracy versus 72.5 for MaTS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Flatten the summed task-matrix spectrum with isotropic scaling (Iso-C) | Task Arithmetic merging · Vision task sets | accuracy | Helped |
| Add task-specific directions to the common subspace (Iso-CTS) | Iso-C · Vision, 14- and 20-task sets | absolute accuracy: up to 2.8% | Helped |
| Add task-specific directions to the common subspace (Iso-CTS) | Iso-C · T0 mixture 7- and 8-task sets | average absolute accuracy | No clear effect |
| Interpolate the spectrum from Task Arithmetic toward Iso-C (ablation) | Task Arithmetic merging · ViT-B/16 vision tasks | accuracy | Helped |
| Replace low-significance common directions with task-specific directions (ablation) | Iso-C · Vision tasks | performance | Helped |
| Use common-subspace fraction k/r = 1 instead of the best reported fraction (ablation) | Iso-CTS · ViT-B/16, 20 tasks | accuracy: 2.8% drop | Hurt |
| Vary the common-subspace fraction k/r (ablation) | Iso-CTS · ViT-B/16, 20 tasks | accuracy: less than 0.5% | Helped |
| Remove isotropic scaling (ablation) | Iso-CTS · ViT-B/16, 20 tasks with a common subspace | NAI | Hurt |
| Remove isotropic scaling (ablation) | Iso-CTS · ViT-B/16, 20 tasks with k/r = 0 | NAI | No clear effect |
| Flatten the singular-value spectrum of each task matrix individually (ablation) | Individual fine-tuned task models · Vision tasks | average performance: 3.3% drop | Hurt |
| Increase subspace alignment using Iso-C instead of Task Arithmetic | Task Arithmetic merging · Vision tasks | interference (mean L1 distance) | Helped |
| Increase the average subspace alignment ratio | Task Arithmetic merging · Task-specific vision datasets | normalized accuracy improvement: ρ_TA = 0.94 | Helped |
Surprising findings
- Cosine similarity between task vectors and the merged TA vector shows no clear correlation with normalized accuracy improvement, while subspace alignment correlates strongly (Pearson 0.94).
- Applying spectrum flattening to individual task matrices reduces average single-task performance by 3.3%, so Iso-C's gains come from mitigating interference from summing tasks.
Limitations
- The common subspace is determined by Task Arithmetic, which may be suboptimal.
- Iso methods are not yet adapted to exploit the low-rank structure of LoRA task matrices for efficiency.
- Only static merging is considered; dynamic merging methods are excluded.
Experimental setup
Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN CIFAR100 STL10 Flowers102 OxfordIIITPet
Benchmarks: 8-task vision benchmark 14-task vision benchmark 20-task vision benchmark KnOTS LoRA 8-task benchmark T0 NLP 8 and 7 task subsets
Metrics: absolute accuracy normalized accuracy normalized accuracy improvement subspace alignment ratio
Baselines: Weight Averaging Task Arithmetic TIES-Merging Consensus TA TSV-M KnOTS-TIES KnOTS-DARE-TIES Fisher Merging RegMean MaTS PCB CART
Builds on: Task Arithmetic Task Singular Vectors KnOTS MaTS
Closest papers by meaning
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.90)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.90)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.90)
- Training-Free Pretrained Model Merging (CVPR 2024 · similarity 0.90)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.89)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.