Research map / Large language models / Instruction and fine-tuning

GoRA: Gradient-driven Adaptive Low Rank Adaptation

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Low-rank adaptation variants · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GoRA, which uses pre-training gradients to allocate per-layer LoRA ranks and to initialize the B matrix via a pseudo-inverse of the gradient, without modifying pre-trained weights.

Problem

LoRA's performance depends on rank selection and initialization, but existing adaptive-rank methods inflate trainable parameters and nonzero initializations manipulate pre-trained weights, creating a training-inference gap.

Approach

Treat LoRA adapters as gradient compressors: compute N-batch accumulated gradients before training, allocate ranks per layer by a sensitivity-based importance score (avg |W ⊙ G|) under a fixed parameter budget, and initialize B = -(A0^T A0)^{-1} A0^T G with A0 kept Kaiming-random, so W0 is never modified.

Key results

• On Llama-3.1-8B-Base, GoRA reaches 72.91 on GSM8k vs LoRA-GA 71.39 and LoRA 67.78; GoRA with r_ref=128 reaches 75.74 GSM8k and 52.03 HumanEval, above full fine-tuning (73.69, 51.63).
• On T5-Base GLUE, GoRA averages 87.96 vs full fine-tuning 87.91 and LoRA-GA 87.77.
• On CLIP-ViT-B/16 across seven image tasks, GoRA averages 89.47 vs LoRA-GA 88.51 and full fine-tuning 88.06.

Measured effects reported in the paper

Surprising findings

• GoRA with a larger reference rank (r_ref=128) on Llama-3.1-8B-Base has lower MTBench (5.82) than r_ref=8 (6.34), so higher rank does not uniformly help the dialogue score.
• Relative gradient reconstruction error is about 11%, yet downstream gains persist, suggesting initialization quality matters beyond exact gradient approximation.

Limitations

• Evaluation limited to models up to 8B (plus a 32B timing test) and relatively small datasets; authors note larger models are untested.
• Initialization of A is still random; authors suggest pre-trained-feature-based A initialization as future work.
• Hyperparameter gamma and rank range (r_min, r_max) require tuning, and gains on MTBench were not consistent (GoRA 6.34 vs LoRA+ 6.35).

Experimental setup

Datasets: MetaMathQA Code-Feedback WizardLM GLUE (MNLI, SST-2, CoLA, QNLI, MRPC) Stanford-Cars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN

Benchmarks: GLUE MTBench GSM8k HumanEval Stanford-Cars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN

Metrics: accuracy pass@1 regex-extracted accuracy average score (0-10)

Baselines: LoRA rsLoRA DoRA LoRA+ OLoRA PiSSA LoRA-GA AdaLoRA LoRA-Pro Full fine-tuning

Builds on: LoRA rsLoRA LoRA-GA AdaLoRA PiSSA

Closest papers by meaning

• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.93)
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.93)
• Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.93)
• Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.92)
• LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.92)