Research map / Architectures and efficiency / Pruning and efficient tuning
LoRA-GA: Low-Rank Adaptation with Gradient Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA-GA, which initializes LoRA adapters using a low-rank approximation of the full fine-tuning gradient so that early LoRA updates mimic full fine-tuning and converge faster.
Problem
LoRA reduces per-iteration compute and memory for fine-tuning large pretrained models, but it converges more slowly and underperforms full fine-tuning.
Approach
Initialize the LoRA low-rank matrices via a gradient approximation: decompose the gradient of the first full fine-tuning step with SVD so that the initial LoRA update approximates full-parameter gradient descent, improving convergence speed and final accuracy without changing the LoRA architecture or inference cost.
Key results
- LoRA-GA is reported to converge faster and match or exceed full fine-tuning accuracy on the evaluated tasks, per the paper's claims.
- Improvements over vanilla LoRA and other LoRA variants on GLUE, GSM8K, MATH, HumanEval and MBPP.
- Gradient-based initialization adds no extra inference cost because the LoRA architecture is unchanged.
Limitations
- Provided text is truncated; limitations section not visible in the extract.
- Initialization requires an initial full-gradient computation, adding one-time overhead.
Experimental setup
Datasets: GLUE GSM8K MATH HumanEval MBPP MT-Bench
Benchmarks: GLUE GSM8K MATH HumanEval MBPP MT-Bench
Metrics: accuracy pass@1 MT-Bench score
Baselines: LoRA full fine-tuning PiSSA LoRA-FA DoRA rsLoRA LoRA+ MiLoRA
Builds on: LoRA PiSSA LoRA-FA
Closest papers by meaning
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.94)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.93)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.93)
- LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation (ICLR 2025 · similarity 0.92)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.92)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.