Research map / Architectures and efficiency / Pruning and efficient tuning

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Pico, a data-free pre-merge calibration that downscales over-shared output-side LoRA B directions and rescales the merged update, improving Task Arithmetic, TIES, and TSV-M LoRA merging on Llama-3.1-8B across four domains.

Problem

Merging independently trained LoRA adapters often degrades performance because merge interference is concentrated in shared directions; existing merge methods treat the update BA as one object without distinguishing its two factors.

Approach

Empirically, the output-side matrix B shares far more directions across tasks than the input-side A; Pico calibrates only B before merging by downscaling shared-basis components according to a sharing score, then rescales the merged update's Frobenius norm to the average source magnitude.

Key results

• Pico improves the overall average over no-calibration by 3.4 points for Task Arithmetic (0.4093 to 0.4430), 4.7 for TIES (0.3859 to 0.4328), and 8.3 for TSV-M (0.3473 to 0.4305).
• Pico with Task Arithmetic (0.4430) exceeds a jointly trained multi-task LoRA (0.3688) by 7.4 points overall.
• Calibrating only B gives the best ablation result (0.4430), while calibrating A (0.3916) or the full BA (0.3743) lowers the overall average versus Task Arithmetic alone (0.4093).

Surprising findings

• Calibrating the input-side A matrix or the full update BA hurts performance, and removing magnitude restoration after B calibration drops the overall average to 0.3908.
• The joint multi-task LoRA underperforms several merged adapters, with overall average 0.3688 versus 0.4430 for Pico.

Limitations

• Evaluation limited to four domain adapters on Llama-3.1-8B at rank 16 in the main setting, plus a Qwen3-4B-Base transfer check.
• Gains on some benchmarks are mixed; e.g., Pico underperforms on some individual benchmarks such as GSM8K and FinanceBench for TIES and TSV-M.
• Motivation analysis rests on overlap and effective-rank diagnostics whose link to merge failure is argued rather than causally tested.

Experimental setup

Datasets: MetaMathQA Magicoder-Evol-Instruct-110K ODA-Fin-SFT-318k Medical-Reasoning-SFT-Trinity-Mini Lawyer-Instruct

Benchmarks: GSM8K MATH HumanEval MBPP FinanceBench ConvFinQA PubMedQA MedQA-USMLE

Metrics: accuracy average score across benchmarks overlap score effective rank

Baselines: Task Arithmetic TIES TSV-M DARE DELLA KnOTS Core Space Joint Multi-Task LoRA

Builds on: LoRA Task Arithmetic TIES TSV-M

Closest papers by meaning

• When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation (NeurIPS 2026 · similarity 0.90)
• Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
• IterIS: Iterative Inference-Solving Alignment for LoRA Merging (CVPR 2025 · similarity 0.88)
• Accurate and Efficient Low-Rank Model Merging in Core Space (NeurIPS 2025 · similarity 0.88)
• Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.86)
• No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.86)