Research map / Architectures and efficiency / Pruning and efficient tuning
BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BSLoRA, a multi-LoRA PEFT method that splits each LoRA update into local, intra-layer shared, and inter-layer shared components, with three shape-transformation methods (Slice Sharing, Gate Transformation, Kronecker Extension), reaching ~45% of standard LoRA's parameters with small accuracy gains on Llama.
Problem
Standard LoRA adds many trainable and stored parameters when many task-specific adapters are deployed together, and existing sharing methods capture either local or global structure but not both, while shared parameters must also fit modules of different shapes.
Approach
Decompose the LoRA update into three parallel sub-LoRAs (module-local, shared within a layer, shared across layers) with separate ranks, and adapt shared matrices to differing module shapes via slicing, low-rank gating, or Kronecker extension with module-specific kernels.
Key results
- BSLoRA variants use about 44.6% of standard LoRA's parameters on average, reported as ~56% reduction in the conclusion; Llama 3-8B commonsense average 72.70 (SS) vs 72.42 for LoRA r=8.
- On MMLU, BSLoRA (SS) on Llama 1-7B reaches 38.98 five-shot average vs 37.15 for LoRA; on Llama 1-13B 48.43 (KE) vs 48.39 LoRA five-shot.
- Average gains reported as about 0.33% on commonsense reasoning and 2.08% on MMLU over LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BSLoRA (SS) instead of LoRA r=8 | Llama 1-7B · Commonsense Reasoning (8-task Avg.) | average score | Helped |
| Use BSLoRA (SS) instead of LoRA r=8 | Llama 3-8B · Commonsense Reasoning (8-task Avg.) | average score | Helped |
| Use BSLoRA (SS) instead of LoRA r=8 | Llama 1-13B · Commonsense Reasoning (8-task Avg.) | average score | No clear effect |
| Use BSLoRA (SS) instead of LoRA r=8 | Llama 1-7B · MMLU 0-shot | average score | Helped |
| Use BSLoRA (SS) instead of LoRA r=8 | Llama 1-7B · MMLU 5-shot | average score | Helped |
| Use BSLoRA (SS) instead of LoRA r=8 | Llama 3-8B · MMLU 0-shot | average score | Helped |
| Use BSLoRA (GT) instead of LoRA r=8 | Llama 3-8B · MMLU 5-shot | average score | Helped |
| Use BSLoRA (KE) instead of LoRA r=8 | Llama 1-13B · MMLU 0-shot | average score | Helped |
| Use BSLoRA (KE) instead of LoRA r=8 | Llama 1-13B · MMLU 5-shot | average score | No clear effect |
| Increase Gate Transformation shared-weight size (ablation) | BSLoRA (GT), Llama 1-7B · MMLU 0-shot | average score: 36.03; 35.61; 34.70 | Hurt |
| Increase Gate Transformation shared-weight size (ablation) | BSLoRA (GT), Llama 1-7B · GSM8K 5-shot | accuracy: 10.54; 10.77; 11.90 | Helped |
| Vary Kronecker Extension shared-weight size (ablation) | BSLoRA (KE), Llama 1-7B · MMLU 0-shot | average score: 36.47; 35.34; 37.01 | Mixed |
Surprising findings
- Sharing parameters was reported to improve performance over standard LoRA while reducing parameter count, and the paper argues the shared sub-LoRAs do not reduce the overall rank of the combined update.
Limitations
- Gains over LoRA are small (about 0.33% commonsense, 2.08% MMLU) and differences across methods are within noise in many cells.
- Serving efficiency (paging, latency) is discussed but not implemented or benchmarked; the S-LoRA integration is left for future work.
- Evaluation is limited to Llama models, a single Alpaca fine-tuning dataset, and one epoch of training.
Experimental setup
Datasets: Alpaca (cleaned)
Benchmarks: BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA SIQA MMLU GSM8K
Metrics: accuracy average accuracy trainable parameter count and ratio
Baselines: LoRA VeRA VB-LoRA Tied-LoRA ShareLoRA
Builds on: LoRA MultiLoRA VeRA VB-LoRA ShareLoRA
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.92)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.92)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.92)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.92)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.