Research map / Architectures and efficiency / Pruning and efficient tuning
Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA-LEGO, which splits LoRAs into rank-wise minimal semantic units, clusters them with K-means, and rebuilds a merged LoRA from the centroids with norm and variance rescaling.
Problem
Merging multiple LoRA adapters without additional training causes parameter interference from element-wise fusion, which misaligns rank components and creates knowledge conflicts, degrading performance.
Approach
Treat each LoRA rank (a row of A paired with a column of B) as a minimal semantic unit, cluster these units across all LoRAs with K-means into k clusters, and use cluster centroids as the rank-k merged LoRA, with dual reweighting (parameter norm matching and output variance scaling by sqrt(r/k)).
Key results
- On Llama2-7b multi-task merging of seven LoRAs, LoRA-LEGO reaches 62.21 average vs 58.30 for Ties-Merging and 47.17 for Weight Averaging.
- On Llama2-13b, LoRA-LEGO reaches 70.27 average vs 69.61 for Ties-Merging.
- Pruning a single LoRA: retaining 50% of parameters keeps about 92% of original performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoRA-LEGO instead of Ties-Merging for multi-task LoRA merging | Llama2-7B · Multi-task GLUE average | average score: 58.30 -> 62.21 | Helped |
| Use LoRA-LEGO instead of Ties-Merging for multi-task LoRA merging | Llama2-13B · Multi-task GLUE average | average score: 69.61 -> 70.27 | Helped |
| Use LoRA-LEGO instead of the strongest baseline | Llama2-7B · Mixed-task Overall OOD | score: 45.8 -> 51.0 | Helped |
| Use LoRA-LEGO instead of the strongest baseline | Llama2-13B · Mixed-task Overall OOD | score: 52.3 -> 52.8 | Helped |
| Apply parameter reweighting to cluster centroids (ablation) | LoRA-LEGO · Merged LoRA at rank 1r | performance: +5% | Helped |
| Apply parameter reweighting without output reweighting (ablation) | LoRA-LEGO · Merged LoRA at higher ranks | performance | Hurt |
| Apply output reweighting to stabilize variance (ablation) | LoRA-LEGO · Merged LoRA at high rank | performance | Helped |
| Combine parameter reweighting and output reweighting (ablation) | LoRA-LEGO · Merged LoRA across varying ranks | performance | Helped |
| Retain 50% of the parameters through LoRA-LEGO pruning (ablation) | Single LoRA · LoRA pruning | retained performance: 50%; 92% | No clear effect |
| Retain 33% of the parameters through LoRA-LEGO pruning (ablation) | Single LoRA · LoRA pruning | retained performance: 33%; 79% | Hurt |
Surprising findings
- Averaging parameters of two identical LoRAs with random row/column misalignment degrades performance, showing parameter misalignment alone harms merging.
Limitations
- Evaluation limited to Llama-2 7b/13b and GLUE-style classification tasks plus flan-v2 mixed pool.
- Relies on Euclidean K-means distance over MSUs; alternative metrics such as optimal transport are unexplored.
- Several OOD and IID results are mixed; LoRA-LEGO does not win every task (e.g., CoLA, WNLI).
Experimental setup
Datasets: CoLA MNLI MRPC QNLI QQP RTE SST2 SNLI WNLI flan-v2
Benchmarks: GLUE SNLI WNLI flan-v2 mixed-task pool
Metrics: accuracy Rouge-1 Rouge-2 Rouge-L BLEU normalized average performance
Baselines: Weight Averaging Ensemble Task Arithmetic Ties-Merging Selection
Builds on: LoRA Task Arithmetic TIES-Merging LoraRetriever
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.92)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.92)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.92)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.92)
- NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.