Research map / Architectures and efficiency / Pruning and efficient tuning

LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail)

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes the LoRA loss landscape without linearization, showing that under restricted strong convexity/smoothness every second-order stationary point is either a low-rank, small-magnitude global minimizer or a high-rank, large-magnitude spurious point, and argues zero-init and weight decay bias training toward the former.

Problem

Understanding whether LoRA fine-tuning converges to global minima, given the non-convexity of the factorization AB^T and the neural network, without relying on linearization or highly simplified setups.

Approach

Introduces restricted strong convexity (alpha) and restricted smoothness (beta) of the full fine-tuning loss around a low-rank global minimizer, splitting into a special regime (2alpha > beta, all SOSPs are global minima) and a generic regime (2alpha <= beta, spurious SOSPs must be high-rank and far from the minimizer); argues zero init and weight decay bias SGD toward the global minimizer.

Key results

• Global minimizer rank decreases with weight decay on SST-2 (rank 749 at lambda=0 to 1 at lambda=0.1), and CIFAR100 similarly (752 to 1), so low-rank global minima are plausible.
• Estimated RSC/RSM show beta/alpha grows with rank (8.02 at r=8, 18.7 at r=16, 320.8 at r=32), and alpha becomes negative at r=64 so the theory no longer applies.
• A spurious local minimum found from large random initialization exhibits high rank, large norm, and poor test accuracy, while zero-init training converges to a low-rank global minimizer.

Measured effects reported in the paper

Surprising findings

• Restricted strong convexity and smoothness constants become less favorable as LoRA rank increases, with alpha turning negative at rank 64, suggesting smaller ranks have better landscapes.
• The theory's implicit-bias argument is supported only by adapting prior arguments and an informal lemma rather than a rigorous proof.

Limitations

• The guarantees rely on assumptions (existence of a low-rank global minimizer, alpha>0, beta<infinity) that are only verified empirically and locally via Monte-Carlo estimates.
• The implicit bias argument (zero init and weight decay favor global minima) is heuristic; the paper says it is not rigorously proven.
• Experiments are small-scale (RoBERTa-base, ViT on SST-2 and CIFAR100) and the spurious-minimum example is a single illustrative case.

Experimental setup

Datasets: SST-2 CIFAR100

Benchmarks: SST-2 CIFAR100

Metrics: training loss test accuracy minimizer rank minimizer norm RSC constant alpha RSM constant beta

Baselines: NTK-regime LoRA analysis linearized LoRA analysis (complete linearization) two-layer teacher-student LoRA analysis

Builds on: LoRA Burer-Monteiro factorization matrix sensing low-rank optimization

Closest papers by meaning

• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.92)
• The Impact of Initialization on LoRA Finetuning Dynamics (NeurIPS 2024 · similarity 0.92)
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.92)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
• The Primacy of Magnitude in Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
• Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics (ICML 2025 · similarity 0.90)