Research map / Architectures and efficiency / Pruning and efficient tuning
RaSA: Rank-Sharing Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RaSA, a LoRA extension that pools k ranks from every layer into a cross-layer shared rank pool with trainable layer-specific diagonal weighting, raising effective update rank at unchanged parameter count.
Problem
LoRA's low-rank constraint limits its expressive capacity, leaving it behind full fine-tuning on complex tasks like code generation and math reasoning.
Approach
Extract k ranks from each layer's LoRA update and share them across all layers as a pool of L*k ranks, combined with layer-specific trainable diagonal scaling, so each layer's update has rank r-k+Lk without extra parameter overhead; theoretically shown to achieve reconstruction error no greater than LoRA's.
Key results
- On HumanEval+ with Llama-3.1-8B at rank 32, RaSA reaches 59.5% PASS@1 (BEST) vs 57.9% for LoRA.
- On MATH with Mistral-0.3-7B at rank 32, RaSA reaches 31.7%... and 26.1% BEST accuracy vs 21.8% for LoRA.
- RaSA with rank 8 outperforms LoRA with rank 32 on MATH (e.g., Llama 31.4 vs 28.9 at r=16 range), with same or fewer trainable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RaSA partial rank sharing instead of LoRA at rank 32 | Llama-3.1-8B · HumanEval+ | Pass@1 (BEST): 57.9 -> 59.5 | Helped |
| Use RaSA partial rank sharing instead of LoRA at rank 32 | Mistral-0.3-7B · HumanEval+ | Pass@1 (BEST): 50.2 -> 55.7 | Helped |
| Use RaSA partial rank sharing instead of LoRA at rank 32 | Llama-3.1-8B · MATH | ACC (BEST): 28.9 -> 31.7 | Helped |
| Use RaSA partial rank sharing instead of LoRA at rank 32 | Mistral-0.3-7B · MATH | ACC (BEST): 21.8 -> 26.1 | Helped |
| Set shared rank k to 0, eliminating rank sharing (ablation) | RaSA reconstruction · Reconstruction error across Llama-3.1-8B linear modules | reconstruction error | No clear effect |
| Increase shared rank k beyond the small value around r/8 (ablation) | RaSA reconstruction · Reconstruction error across Llama-3.1-8B linear modules | reconstruction error | Hurt |
| Share all ranks across layers (k=r) (ablation) | RaSA · Reconstruction error | reconstruction error | Hurt |
| Increase rank from r=8 to r=32 (ablation) | LoRA · MATH | ACC (BEST): 28.3 -> 28.9 | Helped |
| Increase rank from r=8 to r=32 (ablation) | RaSA · MATH | ACC (BEST): 30.3 -> 31.7 | Helped |
| Increase training data from 25% to 50% (ablation) | LoRA · MATH | LAST accuracy | No clear effect |
| Increase training data from 25% to 50% (ablation) | RaSA · MATH | LAST accuracy | Helped |
| Use RaSA instead of LoRA | Llama-3.1-70B and Mixtral-8×7B · MATH | LAST accuracy | Helped |
Surprising findings
- Sharing all ranks across layers (k=r, as in VeRA/Tied-LoRA) yields reconstruction error that can exceed LoRA's, suggesting fully shared methods are sub-optimal.
- Directly increasing LoRA rank gives only marginal math gains while doubling parameters.
Limitations
- Theoretical guarantee is about reconstruction error of matrices, not directly about downstream performance.
- Evaluation limited to two 7-8B models for main results and a few large-scale runs; alpha was changed for math due to convergence issues.
- Requires all layers to share the same dimensionality.
Experimental setup
Datasets: Magicoder-Evol-Instruct-110k MetaMathQA GSM8K MATH
Benchmarks: HumanEval+ MATH HellaSwag WinoGrande ARC-Challenge
Metrics: Pass@1 Pass@10 accuracy reconstruction error training time
Baselines: LoRA MoRA VeRA AdaLoRA PriLoRA OLoRA FFT
Builds on: LoRA AdaLoRA PriLoRA VeRA MoRA
Closest papers by meaning
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.91)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.90)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.