Research map / Architectures and efficiency / Pruning and efficient tuning
LoRA+: Efficient Low Rank Adaptation of Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that standard LoRA uses the same learning rate for adapter matrices A and B, which is provably inefficient at large width, and proposes LoRA+ with a fixed larger learning rate ratio for B than A.
Problem
Standard LoRA finetuning sets the same learning rate for the two low-rank adapter matrices A and B, and there is no principled guidance for choosing the learning rate; at large width this prevents efficient feature learning.
Approach
Using infinite-width scaling arguments, the paper shows that efficient LoRA feature learning requires eta_B / eta_A = Theta(n) (eta_A ~ n^-1, eta_B ~ 1), and proposes LoRA+ which fixes a ratio lambda = eta_B/eta_A > 1 (empirically 2^1 to 2^4) and tunes only eta_A, keeping the cost of standard LoRA.
Key results
- On Llama-7b finetuned on flan-v2, LoRA+ with eta_B >> eta_A gives roughly 1.3% MMLU gain over the best equal-learning-rate setting.
- The abstract reports 1%-2% improvements in performance and up to ~2X finetuning speedup at the same computational cost as LoRA.
- On RoBERTa-base and GPT-2 GLUE finetuning, the best test accuracy is consistently achieved with eta_B >> eta_A, with larger gaps on harder tasks such as MNLI and QQP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Set different learning rates for A and B, using a fixed ratio | LoRA fine-tuning · Multiple language-model tasks | performance: 1%-2% | Helped |
| Set different learning rates for A and B, using a fixed ratio | LoRA fine-tuning · Multiple language-model tasks | finetuning speed: up to ∼2X SpeedUp | Helped |
| Set η_B ≫ η_A instead of η_A = η_B | LoRA fine-tuning · MMLU | accuracy: roughly 1.3% | Helped |
| Use equal learning rates for A and B | LoRA fine-tuning · MMLU | accuracy: roughly 1.3% | Hurt |
| Use η_B ≫ η_A instead of η_A ≈ η_B | RoBERTa-base and GPT-2 LoRA · MNLI and QQP | test accuracy | Helped |
| Use η_B ≫ η_A instead of η_A ≈ η_B | RoBERTa-base LoRA · SST2 and QNLI | test accuracy | Mixed |
| Use η_B ≫ η_A instead of η_A = η_B | Llama-7b LoRA · MNLI | test accuracy | No clear effect |
| Use η_A = η_B | Adam-trained LoRA · Infinite-width feature learning | feature-update efficiency | Hurt |
| Set η_A = Θ(n^{-1}) and η_B = Θ(1) | LoRA fine-tuning · Infinite-width feature learning | feature-update efficiency | Helped |
| Use Init 1 instead of Init 2 (ablation) | Llama-7b LoRA fine-tuned on flan-v2 · MMLU | accuracy: 0.6% | Helped |
| Use Init 1 instead of Init 2 (ablation) | Llama-7b LoRA fine-tuned on flan-v2 · flan-v2 | optimal η_B / η_A ratio: twice as large | Mixed |
| Set η_B > η_A | Llama-7b LoRA fine-tuned on flan-v2 · flan-v2 training loss | training loss | Hurt |
Surprising findings
- Optimal learning rate ratio depends on initialization and model: about 2^4 for RoBERTa with init 2, 2^2-2^3 with init 1, and 2^1-2^2 for Llama.
Limitations
- The optimal ratio lambda = eta_B/eta_A is task- and model-dependent and not given by the asymptotic theory, which only gives Theta(n) scaling.
- Theoretical results are at a physics-level of rigor with simplifying assumptions (single sample, single trainable layer, gradient assumption for Adam).
- Experiments use limited compute, e.g. shorter sequence length (T=128) and fewer epochs than prior reported setups.
Experimental setup
Datasets: MNLI QQP SST2 QNLI flan-v2 synthetic sin dataset
Benchmarks: GLUE MMLU MNLI QQP SST2 QNLI
Metrics: test accuracy train loss test loss speedup
Baselines: LoRA with equal learning rates (eta_A = eta_B)
Builds on: LoRA Maximal Update Parametrization (muP) Adam Tensor Programs
Closest papers by meaning
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.93)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.92)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.92)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.92)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.