Research map / Architectures and efficiency / Pruning and efficient tuning
Unveiling LoRA Intrinsic Ranks via Salience Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SalientLoRA, an adaptive LoRA rank allocation method that scores singular-value salience over a time series using orthogonality-weighted magnitudes and a correlation-based influence-domain graph, with an adaptive time-window to speed up pruning.
Problem
LoRA uses a fixed rank, and AdaLoRA's importance measure ignores the dominant role of singular values and training fluctuations, while its rank-allocation space is limited by fine-tuning cost.
Approach
Measure singular-value salience across a time series by combining orthogonality-weighted singular value magnitudes with an influence-domain score from a Pearson-correlation and regression-slope dependency graph (de-cycled via DFS), and adapt the time-window size so pruning is fast early and cautious later, allowing a larger initial rank.
Key results
- SalientLoRA outperforms AdaLoRA by 0.96% (rt=144) and 1.17% (rt=276) average on GLUE with DeBERTaV3-base.
- On MT-Bench, SalientLoRA scores 5.28 (LLaMA-7B) and 5.42 (LLaMA2-7B), above Full FT at 5.12 and 5.25.
- With expanded initial rank, AdaLoRA runtime rises about 108% on average while SalientLoRA rises only 7% to 9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SalientLoRA instead of the strongest baseline by average score | DeBERTaV3-base · GLUE, rt=144 | average score: 88.88 -> 89.79 | Helped |
| use SalientLoRA instead of the strongest baseline by average score | DeBERTaV3-base · GLUE, rt=276 | average score: 89.46 -> 90.23 | Helped |
| use SalientLoRA instead of Full FT | T5-base · CNN/DailyMail | ROUGE-1: 42.05 -> 43.47 | Helped |
| use SalientLoRA instead of Full FT | T5-base · CNN/DailyMail | ROUGE-2: 20.34 -> 20.52 | Helped |
| use SalientLoRA instead of Full FT | T5-base · CNN/DailyMail | ROUGE-L: 39.40 -> 40.91 | Helped |
| use SalientLoRA instead of the strongest fine-tuning baseline | LLaMA-7B · MT-Bench | score: 5.17 -> 5.28 | Helped |
| use SalientLoRA instead of the strongest fine-tuning baseline | LLaMA2-7B · MT-Bench | score: 5.31 -> 5.42 | Helped |
| add orthogonality-aware magnitude weighting (ablation) | SalientLoRA · GLUE | average score: 89.31 -> 89.79 | Helped |
| add influence-domain measurement (ablation) | SalientLoRA · GLUE | average score: 88.99 -> 89.79 | Helped |
| increase initial rank (ablation) | AdaLoRA · CoLA, RTE, STS-B, and MRPC | performance: +0.39% - 1.44% | Helped |
| increase initial rank (ablation) | SalientLoRA · CoLA, RTE, STS-B, and MRPC | performance: +0.55% - 1.5% | Helped |
| increase initial rank (ablation) | AdaLoRA · CoLA, RTE, STS-B, and MRPC | fine-tuning time: 108% on average | Hurt |
Surprising findings
- SalientLoRA with LoRA-style low-rank adapters surpasses full fine-tuning on MT-Bench for LLaMA and LLaMA2 and on CNN/DailyMail summarization.
Limitations
- Gains on NLU and NLG are modest (under about 2%), and the paper reports no limitations section beyond checklist answers.
- The MT-Bench evaluation relies on GPT-4 judging of 80 questions, and the instruction-tuning experiments use only a single setup.
Experimental setup
Datasets: GLUE XSum CNN/DailyMail Alpaca
Benchmarks: GLUE XSum CNN/DailyMail MT-Bench
Metrics: Matthews correlation coefficient Spearman correlation accuracy ROUGE-1 ROUGE-2 ROUGE-L MT-Bench score runtime
Baselines: Full FT Adapter AdapterFusion LoRA SoRA AdaLoRA DoRA
Builds on: LoRA AdaLoRA DoRA SoRA
Closest papers by meaning
- AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.91)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.91)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.90)
- RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models (NeurIPS 2025 · similarity 0.90)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.