Research map / Architectures and efficiency / Pruning and efficient tuning
AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AltLoRA, an alternating-projection LoRA optimizer that approximates the full gradient by updating A and B alternately with closed-form scaled gradients and low-rank-aligned momentum, keeping LoRA's memory cost.
Problem
LoRA underperforms full fine-tuning; existing gradient-approximation methods such as LoRA-Pro depend on an ambiguous auxiliary variable, need a Sylvester equation, and require full k×d memory when incorporating momentum.
Approach
Alternately update A then B, each step solving a least-squares projection of the full gradient onto the current low-rank factor, and realign momentum to the new low-rank subspace via a closed-form projection, so that momentum is handled inside the low-rank space with O(kr+dr) memory.
Key results
- On Llama-3.1-8B with rank 8, AltLoRA reaches 74.49 on GSM8K vs 66.11 for LoRA and 73.31 for Full FT; AltLoRA+ reaches 76.91 on GSM8K.
- AltLoRA+ scores 50.10 on HumanEval vs 40.31 for LoRA and 50.81 for Full FT, and 6.16 on MT-Bench.
- On GLUE with T5-base, AltLoRA+ averages 88.30, the best average among compared methods, with the highest MRPC score (87.25).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LoRA with AltLoRA at rank 8 | Llama-3.1-8B-Base fine-tuning · MT-Bench, GSM8K, HumanEval | benchmark score: +0.5 on MT-bench, 8.38 on GSM8K and 3.1 on HumanEval | Helped |
| use AltLoRA+ instead of LoRA-Pro | Llama-3.1-8B-Base fine-tuning · GSM8K and HumanEval | benchmark score | Helped |
| use AltLoRA+ instead of other fine-tuning methods | T5-base · GLUE average | accuracy | Helped |
| increase AltLoRA rank from 8 to 32 or 128 (ablation) | AltLoRA on Llama-3.1-8B-Base · MT-Bench, GSM8K, HumanEval | benchmark score | Mixed |
| increase AltLoRA+ rank from 8 to 32 or 128 (ablation) | AltLoRA+ on Llama-3.1-8B-Base · MT-Bench, GSM8K, HumanEval | benchmark score | Mixed |
| use scaling factor α=16 instead of α=8 or α=32 (ablation) | LoRA, AltLoRA, and AltLoRA+ · GSM8K | accuracy | Helped |
Surprising findings
- AltLoRA (first-order momentum) scores below LoRA-Pro and Full FT on MT-Bench-type comparisons, with AltLoRA+ needed to surpass them on several tasks.
- Reported AltLoRA GLUE average (87.55) equals LoRA-Pro's and is below PiSSA's 88.28, so gains are not uniform across tasks.
Limitations
- Transformation invariance is not preserved under second-order momentum (AltLoRA+), as acknowledged by the authors.
- Gains over baselines on GLUE are mixed; AltLoRA alone has an average equal to LoRA-Pro and below PiSSA.
- Results rely on a grid-searched learning rate and three seeds; only a few models and tasks are evaluated.
Experimental setup
Datasets: WizardLM MetaMathQA CodeFeedback GLUE (MNLI, SST-2, CoLA, QNLI, MRPC)
Benchmarks: MT-Bench GSM8K HumanEval GLUE
Metrics: MT-Bench score exact match accuracy pass@1 accuracy
Baselines: Full FT LoRA PiSSA rsLoRA LoRA+ DoRA AdaLoRA LoRA-GA LoRA-Pro LoRA-Rite
Builds on: LoRA LoRA-Pro LoRA-Rite LoRA+ scaled gradient descent
Closest papers by meaning
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.93)
- RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models (NeurIPS 2025 · similarity 0.93)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.93)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.92)
- LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.