Research map / Architectures and efficiency / Pruning and efficient tuning

RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RefLoRA, which refactors LoRA's low-rank factors per step by choosing the SPD matrix that minimizes a loss upper bound (closed-form via matrix geometric mean), plus a cheaper scalar variant RefLoRA-S, evaluated on GLUE, commonsense reasoning, and subject-driven image generation.

Problem

LoRA suffers from slow, unstable convergence and performance gaps because its non-unique low-rank factorization yields inconsistent and imbalanced weight updates between A and B.

Approach

Characterize all equivalent LoRA factorizations by an invertible matrix P, so weight updates depend only on the SPD matrix S = PP^T; minimize a quadratic upper bound on the loss over S, which has a closed-form solution (matrix geometric mean of (A^TA)^{-1} and B^TB) that balances A and B, and apply it via a preconditioned update that avoids transforming Adam moments.

Key results

• On GLUE with DeBERTaV3-base (r=8), RefLoRA reaches 89.52 average, the best among compared methods (LoRA 88.50, DoRA 88.98, LoRA-Pro 88.94).
• On commonsense reasoning with LLaMA2-7B (r=32), RefLoRA averages 80.78 versus DoRA 79.7 and LoRA-RITE 80.10.
• RefLoRA incurs 88.5% of LoRA's throughput at +132MB GPU memory, while RefLoRA-S retains 98.7% throughput at under 1MB extra memory.

Measured effects reported in the paper

Surprising findings

• Vanilla LoRA diverges at the higher learning rate in matrix factorization, while RefLoRA remains stable and even outperforms ScaledGD.

Limitations

• Evaluation limited to models up to 8B parameters due to compute; 27B/30B scaling deferred to appendix/future work.
• Relies on Lipschitz-smoothness and rank-preservation assumptions, with smoothness constant L treated as a tuned hyperparameter.
• RefLoRA-S only guarantees a weaker balance and does not give consistent weight updates across factorizations.

Experimental setup

Datasets: GLUE BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA DreamBooth

Benchmarks: GLUE commonsense reasoning (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OBQA) DreamBooth fine-tuning loss

Metrics: accuracy Matthews correlation Pearson correlation training loss throughput GPU memory

Baselines: LoRA DoRA AdaLoRA LoRA-Pro LoRA-RITE PrecLoRA NoRA+ ScaledGD

Builds on: LoRA ScaledGD Adam PiSSA LoRA-Pro

Closest papers by meaning

• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.93)
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.93)
• GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.92)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.91)
• LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.91)
• Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.91)