Research map / Architectures and efficiency / Pruning and efficient tuning

LoRA Training in the NTK Regime has No Spurious Local Minima

ICML 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretically analyzes LoRA fine-tuning in the NTK regime, showing a low-rank full fine-tuning solution exists (rank r with r(r+1)/2 ≤ KN), that LoRA with r(r+1)/2 > KN has no spurious local minima under an ε-perturbation, and that the resulting low-rank solution generalizes via Rademacher bounds.

Problem

Theoretical understanding of why LoRA fine-tuning is trainable (finds good solutions despite nonconvex quadratic parameterization) and generalizes, which prior work has not addressed.

Approach

Treats LoRA's factorization as a Burer–Monteiro-style quadratic parameterization of a nuclear-norm-regularized linearized (NTK) problem; shows a low-rank minimizer exists via dimension counting, and uses a generic positive semidefinite perturbation plus Sard's theorem to show that rank r with r(r+1)/2 > KN yields no spurious SOSPs, so (S)GD converges to near-global low-rank solutions.

Key results

• Full fine-tuning in the NTK regime admits a rank-r solution with r(r+1)/2 ≤ KN.
• With LoRA rank satisfying r(r+1)/2 > KN, under ε-perturbation all SOSPs are ε-approximate global minimizers; with N=32 and K=2 the rank threshold is r≥11.
• Experiments on linearized RoBERTa-base, ViT, and wav2vec2 show training losses converge to the same global value for LoRA and full fine-tuning, with differing convergence rates.

Measured effects reported in the paper

Surprising findings

• Lower LoRA rank does not change the attainable optimum but appears to slow convergence, suggesting unfavorable plateaus or saddles despite no spurious local minima.

Limitations

• Analysis restricted to the NTK/linearized regime; does not cover feature learning.
• Global-minimum result requires a generic perturbation P; the unperturbed P=0 case is not directly covered.
• Rank lower bounds are upper bounds only; the paper does not prove the minimum rank is necessary.

Experimental setup

Datasets: SST-2 MR CR QNLI Subj QQP bean disease dataset SUPERB

Benchmarks: SST-2 MR CR QNLI Subj QQP bean disease SUPERB speech

Metrics: training loss vs epochs test accuracy (appendix)

Baselines: full fine-tuning (without LoRA) on linearized model

Builds on: neural tangent kernel Burer-Monteiro factorization LoRA prompt-based fine-tuning (LM-BFF) Ge et al. perturbed matrix factorization landscape results

Closest papers by meaning

• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.92)
• The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.92)
• Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.92)
• LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.91)
• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)