Research map / Large language models / Instruction and fine-tuning

Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Low-rank adaptation variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Re-evaluates nine LoRA variants plus vanilla LoRA with extensive learning-rate, batch-size, rank and duration sweeps, finding that properly tuned methods reach similar peak performance and that optimal learning rates track the largest Hessian eigenvalue.

Problem

Advanced LoRA variants report gains over vanilla LoRA under fixed or narrowly tuned hyperparameters, so it is unclear whether reported improvements reflect real methodological advantages given the sensitivity of training to hyperparameters.

Approach

Systematic joint hyperparameter sweeps (learning rate dominant, then batch size, rank, duration) across ten LoRA methods show peak accuracy parity, with method-specific optimal learning rate ranges explained by differences in top Hessian eigenvalue at initialization.

Key results

• On Qwen3-0.6B (r=128, B=64), all ten LoRA methods reach accuracies within a 0.84% range once learning rate is tuned; GraLoRA leads LoRA-GA by only 0.02%.
• PiSSA's optimal learning rate is about 10x lower than vanilla LoRA's, and OLoRA and LoRA-GA show up to 100x higher top Hessian eigenvalues at initialization.
• Performance gaps across methods are 0.52% (Gemma-3-1B math), 0.43% and 1.75% (Llama-2-7B math and code), and 1.81% (Llama-2-13B math).

Surprising findings

• Vanilla LoRA frequently matches or outperforms advanced variants once learning rate is properly tuned.
• Surveyed 64 prior LoRA papers: fewer than 30% tuned the learning rate and only one tuned all three of learning rate, batch size, and rank.

Limitations

• Only decoder-only LLMs from 0.6B to 13B were tested; scalability to larger models is unverified.
• Secondary settings (schedulers, warmup, adapter placement) were fixed; only a subset of methods was tuned over batch size and rank.
• Architecture and optimization variants (DoRA, GraLoRA, RandLoRA, LoFT) were not analyzed via Hessian.

Experimental setup

Datasets: MetaMathQA CodeFeedback Alpaca commonsense reasoning training set (15k, 8 subtasks)

Benchmarks: GSM8K MATH HumanEval MBPP IFEval commonsense reasoning suite

Metrics: accuracy prompt-level strict accuracy top Hessian eigenvalue ratio

Baselines: LoRA DoRA PiSSA MiLoRA Init[AB] OLoRA LoRA-GA GraLoRA RandLoRA LoFT

Builds on: LoRA PiSSA DoRA LoRA-GA LoFT

Closest papers by meaning

• Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.93)
• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
• GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.93)
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.93)
• SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.93)
• Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.92)