Research map / Optimization / Optimizers
LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that under gradient descent LoRA adapters align with top singular subspaces of the one-step full fine-tuning gradient, and proposes LoRA-One, a spectral initialization from that gradient (with preconditioning) that improves over LoRA and gradient-alignment variants on GLUE, math, and code tasks.
Problem
LoRA's optimization dynamics are nonlinear and non-convex, and it is unclear which subspaces LoRA converges to or how theory can guide better initialization for parameter-efficient fine-tuning of LLMs.
Approach
Theory shows LoRA GD iterates align with the top singular subspaces of the one-step full gradient; initializing A and B from the SVD of that gradient (spectral initialization) gives immediate alignment and small initial error, and preconditioned GD removes dependence on the condition number of the shift.
Key results
- On GLUE subset with T5-base (rank 8), LoRA-One averages 88.73% vs 82.72% for LoRA and 88.44% for LoRA-Pro.
- On LLaMA 2-7B, LoRA-One reaches 28.66 PASS@1 on HumanEval vs 25.85 for LoRA.
- LoRA-One on GSM8K direct prompting gets 60.44% vs 59.26% for LoRA, with near-identical training time and memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-step full-gradient spectral initialization instead of LoRA-Pro initialization | LoRA fine-tuning · MNLI | accuracy: 86.03$_{\pm0.19}$ -> 85.89$_{\pm 0.08}$ | Hurt |
| Use one-step full-gradient spectral initialization instead of LoRA-Pro initialization | LoRA fine-tuning · QNLI | accuracy: 93.42$_{\pm0.05}$ -> 93.37$_{\pm0.02}$ | Hurt |
| Use one-step full-gradient spectral initialization instead of LoRA-Pro initialization | LoRA fine-tuning · GLUE subset average | accuracy: 88.44 -> 88.73 | Helped |
| Use one-step full-gradient spectral initialization instead of LoRA initialization | LoRA fine-tuning · GSM8K direct prompting | accuracy: 59.26$_{\pm 0.99}$ -> 60.44$_{\pm 0.17}$ | Helped |
| Use one-step full-gradient spectral initialization instead of LoRA initialization | LoRA fine-tuning · GSM8K 8-shot CoT | accuracy: 53.36$_{\pm 0.77}$ -> 55.88$_{\pm 0.44}$ | Helped |
| Use one-step full-gradient spectral initialization instead of LoRA initialization | LoRA fine-tuning · MMLU | accuracy: 45.73$_{\pm 0.30}$ -> 47.24$_{\pm 0.20}$ | Helped |
| Use one-step full-gradient spectral initialization instead of LoRA-GA initialization | LoRA fine-tuning · HumanEval | PASS@1: 26.95$_{\pm 1.30}$ -> 28.66$_{\pm 0.39}$ | Helped |
| Add preconditioning to LoRA-One updates | LoRA-One · SST-2, overly large learning rate (\SI{1e-3}{}) | test accuracy | Helped |
| Add preconditioning to LoRA-One updates | LoRA-One · SST-2, moderate learning rate (\SI{2e-4}{}) | test accuracy | No clear effect |
| Add preconditioning to LoRA-One updates | LoRA-One · SST-2, small learning rates (\SI{1e-4}{} to \SI{1e-5}{}) | test accuracy | Helped |
Surprising findings
- Updating the base model with only a rank-r approximation of the one-step gradient (computed on a small batch) matches or beats LoRA on small tasks such as CoLA and MRPC, in under one second.
- LoRA-GA's initialization picks singular vectors r+1 to 2r, which the theory suggests misaligns B with the relevant subspace.
Limitations
- Theory covers linear models and a single-hidden-layer ReLU setting with Gaussian or sub-Gaussian data and label-noiseless assumptions.
- Experiments use rank 8 only and report results with a small number of seeds for some settings.
Experimental setup
Datasets: GLUE (MNLI, SST-2, CoLA, QNLI, MRPC) MetaMathQA Code-Feedback Alpaca
Benchmarks: GLUE subset GSM8K MMLU HumanEval
Metrics: accuracy PASS@1 training time memory
Baselines: LoRA LoRA+ P-LoRA PiSSA LoRA-GA LoRA-Pro LoRA-SB
Builds on: LoRA LoRA-GA LoRA-Pro PiSSA LoRA+
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.92)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.92)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.92)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.92)
- AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.