Research map / Architectures and efficiency / Pruning and efficient tuning

GeoLoRA: Geometric integration for parameter efficient fine-tuning

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GeoLoRA, a dynamical low-rank training method for LoRA-style adapters that updates basis and coefficients in parallel with a single backward pass per step, giving rank adaptivity, local optimality, and convergence/error-bound guarantees.

Problem

LoRA-style fine-tuning needs adaptive rank allocation, robustness to hyperparameters, and low compute, while existing rank-adaptive methods lack convergence guarantees or require multiple gradient tapes per step.

Approach

Applies a parallel geometric low-rank integrator (from dynamical low-rank approximation) to USV^T adapters: one gradient evaluation computes K and L updates in parallel, basis augmentation doubles rank, and singular-value truncation adapts rank per layer or globally, following the projected gradient flow on the rank-r manifold.

Key results

• On GLUE with DeBERTaV3-base, GeoLoRA reports the best score on SST-2 (95.98), CoLA (69.03), MRPC (90.10) and STS-B (91.58) among compared methods, using about 0.7-1.2M parameters.
• On ViT-base fine-tuning to CIFAR-100, GeoLoRA (local budgeting) reaches 91.63% with 0.35M parameters versus AdaLoRA's 91.44% with 0.45M.
• On Stable Diffusion DreamBooth, GeoLoRA (tau=0.02) reaches validation loss 0.242 with 2.6M parameters versus LoRA (r=5) at 0.275 with 3.0M parameters.

Measured effects reported in the paper

Surprising findings

• The paper reports GeoLoRA being less sensitive to high learning rates than AdaLoRA, where AdaLoRA diverges (nan) in some settings.

Limitations

• Convergence and error-bound guarantees rely on assumptions (L-smoothness, bounded gradients, small residual) and only give local optimality.
• Initial S0 inverse in the first iteration is approximated by identity, which is inconsistent with the optimal dynamics for that step.
• Reported GLUE gains are small and within run-to-run variance in several tasks; several baselines' numbers are taken from other papers.

Experimental setup

Datasets: GLUE CIFAR-10 CIFAR-100 Tiny-ImageNet DreamBooth

Benchmarks: GLUE CIFAR-10 CIFAR-100 Tiny-ImageNet DreamBooth

Metrics: accuracy Matthews correlation F1 Pearson/Spearman correlation validation loss iterations per second trainable parameters

Baselines: Full FT BitFit Houlsby adapter Pfeiffer adapter LoRA AdaLoRA DLRT

Builds on: LoRA AdaLoRA DLRT dynamical low-rank approximation

Closest papers by meaning

• RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.90)
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.90)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.90)
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.89)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.89)
• GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.89)