Research map / Architectures and efficiency / Pruning and efficient tuning
Model merging with SVD to tie the Knots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KnOTS, a data- and gradient-free SVD-based alignment of LoRA task-updates across models that makes existing gradient-free merging methods (TIES, DARE-TIES, TA) work better, and introduces a joint-task benchmark for merged-model generality.
Problem
Gradient-free model merging transfers poorly to LoRA-finetuned models because their task-updates are weakly aligned in representation space, unlike full-rank finetuned models.
Approach
Concatenate the LoRA task-updates of a layer column-wise, take a joint SVD, and merge only the per-task right-singular components V^T in the shared aligned basis U·Σ, then rebuild the merged update as UΣV^(merged)T; this is plug-and-play with existing merging methods.
Key results
- KnOTS-TIES improves TIES by 4.3% average normalized accuracy on eight ViT-B/32 LoRA models (68.0 vs 63.7).
- On the joint-task Union setting, KnOTS-TIES improves Hits@1 by up to 3.2% over baselines (46.8 Hits@1 vs 43.6 for TIES).
- On six Llama-3-8B NLI LoRA models, KnOTS-TIES outperforms baseline merging by up to 2.9% average normalized accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply KnOTS alignment with TIES instead of Fisher merging | ViT-B/32 eight-task vision per-task merge · Eight-task vision per-task (Avg) | normalized accuracy (%): 63.9 -> 68.0 | Helped |
| Apply KnOTS alignment with TIES instead of DARE-TIES | ViT-B/32 joint-task merge · Joint-task Union | Hits@1 (%): 44.0 -> 46.8 | Helped |
| Apply KnOTS alignment with TIES instead of DARE-TIES | ViT-B/32 joint-task merge · Joint-task Union | Hits@3 (%): 66.4 -> 68.1 | Helped |
| Apply KnOTS alignment with TIES instead of DARE-TIES | ViT-B/32 joint-task merge · Joint-task Union | Hits@5 (%): 75.1 -> 76.3 | Helped |
| Concatenate task-updates row-wise instead of column-wise (ablation) | KnOTS-TIES · Eight-task vision per-task (Avg) | normalized accuracy (%): 68.0 -> 65.4 | Hurt |
| Use KnOTS-TIES rather than TIES at LoRA rank 4 (ablation) | ViT-B/32 eight-task vision merge · Eight-task vision per-task | normalized accuracy (%): 58.6% -> 64.6% | Helped |
| Use KnOTS-TIES rather than TIES at full rank (ablation) | ViT-B/32 eight-task vision merge · Eight-task vision per-task | normalized accuracy (%): +4% | Helped |
Surprising findings
- Task-vector orthogonality (cosine similarity) does not reliably predict merge potential, as shown by a toy example where orthogonal task vectors still yield catastrophic merges.
- Concatenating task-updates row-wise instead of column-wise lowers average normalized accuracy by 2.6%.
Limitations
- Improvements are modest in some settings (e.g., KnOTS-DARE-TIES gains only 0.2% on NLI) and joint-task absolute performance remains low.
- Only gradient-free merging methods are studied; the approach is not compared against gradient-based merging beyond Fisher averaging in the appendix.
- Joint-task label synonyms are handled by Hits@k rather than by removing ambiguous labels.
Experimental setup
Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN SNLI MNLI SICK QNLI
Benchmarks: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN SNLI MNLI SICK QNLI
Metrics: normalized accuracy Hits@1 Hits@3 Hits@5
Baselines: RegMean Task Arithmetic TIES DARE-TIES Fisher weight averaging Ensemble
Builds on: task arithmetic TIES-merging DARE RegMean LoRA centered kernel alignment
Closest papers by meaning
- Accurate and Efficient Low-Rank Model Merging in Core Space (NeurIPS 2025 · similarity 0.87)
- IterIS: Iterative Inference-Solving Alignment for LoRA Merging (CVPR 2025 · similarity 0.86)
- Generalizing the Geometry of Model Merging Through Fréchet Averages (NeurIPS 2026 · similarity 0.86)
- PLeaS - Merging Models with Permutations and Least Squares (CVPR 2025 · similarity 0.85)
- Crowded in B-Space: Calibrating Shared Directions for LoRA Merging (NeurIPS 2026 · similarity 0.85)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.