Research map / Architectures and efficiency / Pruning and efficient tuning
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LoRA optimization equals full fine-tuning with a low-rank equivalent gradient, and proposes LoRA-Pro, which adjusts A and B gradients via a closed-form solution (with a Sylvester-equation X) to narrow the gap to full fine-tuning.
Problem
LoRA still underperforms full fine-tuning; the optimization dynamics of LoRA differ from full fine-tuning, so the gap needs to be closed through better optimization of the low-rank matrices.
Approach
LoRA updates correspond to full fine-tuning with a low-rank equivalent gradient sBg^A + sg^BA; LoRA-Pro chooses the A and B gradients that minimize the distance between this equivalent gradient and the true full gradient, yielding a closed-form solution expressed via standard LoRA gradients plus an arbitrary-matrix term X fixed by a Sylvester equation.
Key results
- On GLUE subset with T5-base (rank 8), LoRA-Pro averages 88.44 versus 82.08 for LoRA and 87.91 for full fine-tuning.
- On Llama-2-7B with rank 8, LoRA-Pro reaches 57.57 on GSM8K versus 42.08 for LoRA and 59.36 for full fine-tuning; 22.97 on HumanEval versus 14.76 for LoRA.
- On CLIP-ViT-B/16 image classification across seven datasets, LoRA-Pro averages 89.20, outperforming LoRA (83.46) and full fine-tuning (88.06).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adjust the LoRA A and B gradients using LoRA-Pro | LoRA on T5-base · GLUE average (MNLI, SST2, CoLA, QNLI, MRPC) | average score: 82.08 -> 88.44 | Helped |
| Adjust the LoRA A and B gradients using LoRA-Pro | LoRA on CLIP-ViT-B/16 · StanfordCars, DTD, EuroSAT, GTSRB, RESISC45, SUN397, SVHN | accuracy: +5.74 average | Helped |
| Adjust the LoRA A and B gradients using LoRA-Pro | LoRA on Llama-2-7B · GSM8K | accuracy: 42.08 -> 57.57 | Helped |
| Adjust the LoRA A and B gradients using LoRA-Pro | LoRA on Llama-2-7B · HumanEval | PASS@1: 14.76 -> 22.97 | Helped |
| Adjust the LoRA A and B gradients using LoRA-Pro | LoRA on Llama-2-7B · MT-Bench | first-turn score: 5.61 -> 5.72 | Helped |
| Use full fine-tuning rather than LoRA-Pro | Llama-2-7B · GSM8K | accuracy: 57.57 -> 59.36 | Helped |
| Use full fine-tuning rather than LoRA-Pro | Llama-2-7B · HumanEval | PASS@1: 22.97 -> 35.31 | Helped |
| Use full fine-tuning rather than LoRA-Pro | Llama-2-7B · MT-Bench | first-turn score: 5.72 -> 5.30 | Hurt |
| Increase LoRA-Pro rank from 8 to 128 | Llama-2-7B · GSM8K and HumanEval | accuracy / PASS@1: 57.57±0.50 -> 61.08±0.19; 22.97±0.35 -> 30.28±0.93 | Helped |
| Set X to zero instead of using the Sylvester solution (ablation) | LoRA-Pro · GSM8K | accuracy: 57.57±0.50 -> 31.74±0.69 | Hurt |
| Set X to zero instead of using the Sylvester solution (ablation) | LoRA-Pro · HumanEval | PASS@1: 22.97±0.35 -> 17.28±0.35 | Hurt |
| Use the symmetry solution for X instead of the Sylvester solution (ablation) | LoRA-Pro · GSM8K | accuracy: 57.57±0.50 -> 42.81±0.62 | Hurt |
Surprising findings
- LoRA-Pro with rank 8 slightly outperforms full fine-tuning on the GLUE subset, which the authors attribute to overfitting in full fine-tuning.
- Full fine-tuning underperforms LoRA-Pro and LoRA on MT-Bench, which the authors attribute to overfitting to the WizardLM distribution.
Limitations
- Still relies on the low-rank assumption for the weight update, which may fail in pre-training or with large fine-tuning data.
- Only applicable to LoRA-like structures; not directly applicable to DoRA or FLoRA.
- About 4GB extra GPU memory compared to LoRA.
Experimental setup
Datasets: GLUE WizardLM MetaMathQA CodeFeedback StanfordCars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN
Benchmarks: GLUE MT-Bench GSM8K HumanEval StanfordCars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN
Metrics: accuracy MT-Bench first-turn score pass@1 Matthews/F1 average accuracy
Baselines: Full fine-tuning LoRA rsLoRA LoRA+ PiSSA DoRA AdaLoRA LoRA-GA
Builds on: LoRA rsLoRA LoRA-GA AdamW
Closest papers by meaning
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.94)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.93)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.93)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.93)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.