Research map / Architectures and efficiency / Pruning and efficient tuning

Accurate and Efficient Low-Rank Model Merging in Core Space

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Core Space, a lossless shared low-rank basis for merging LoRA-adapted models so any merging method (TA, TIES, TSV, Iso-C, CART) runs on small r-dimensional-per-task cores instead of full weight matrices.

Problem

Merging LoRA-adapted models is suboptimal when done on low-rank products directly, and alignment-based merging (KnOTS) requires full-size SVDs that are costly for large models.

Approach

Build shared reference bases via SVD of stacked LoRA A and B factors across tasks, project each task's LoRA update into a Tr x Tr core matrix that is exactly reconstructible, then apply any merging operator in this compact aligned space.

Key results

• On Llama 3 8B NLI, TSV in Core Space reaches 94.16% average normalized accuracy, versus 92.55% for TSV in full space, at roughly 280x lower relative merge time.
• On ViT-B/32 vision tasks, TSV + Iso-C in Core Space achieves 76.43% average normalized accuracy, above the prior best of 68.0% (TIES in KnOTS space).
• Core Space improves 8 of 9 merging methods versus full space on ViT-B/32 and provides up to ~600x merging speed-up over KnOTS on Llama 3 8B.

Measured effects reported in the paper

Surprising findings

• Full-space TIES yields merged updates of effective rank ~766 instead of the intended 128, because trimming destroys the low-rank structure.
• In full space, up to 80% of singular values can be truncated from a Task Arithmetic merged model without accuracy loss, whereas in Core Space any truncation hurts.

Limitations

• Core Space dimension Tr grows with number of tasks and rank, so the core cost grows as T^3 r^3 with many tasks or large ranks.
• Evaluated only on 8 vision tasks and 6 NLI tasks with a single LoRA rank setup in main results.

Experimental setup

Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN SNLI MNLI SICK QNLI

Benchmarks: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN SNLI MNLI SICK QNLI

Metrics: normalized accuracy absolute accuracy merging time

Baselines: Task Arithmetic TIES DARE-TIES TSV CART Iso-C KnOTS

Builds on: LoRA KnOTS Task Arithmetic Iso-C TSV

Closest papers by meaning

• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.91)
• The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.90)
• Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.90)
• RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.89)
• LoRA+: Efficient Low Rank Adaptation of Large Models (ICML 2024 · similarity 0.89)