Research map / Architectures and efficiency / Pruning and efficient tuning

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SDS-LoRA, a LoRA reparameterization that multiplies each LoRA factor by the orthonormal bases of the other's subspace (refreshed periodically via QR), so the full gradient is backpropagated without singular-value scaling, improving convergence and narrowing the gap to full fine-tuning.

Problem

LoRA lags full fine-tuning because singular values of the LoRA factors anisotropically scale the backpropagated full gradient, reducing gradient effective rank and worsening gradient approximation and convergence.

Approach

Reparameterize the weight update as s(Q_B A + B Q_A^T), where Q_A and Q_B are orthonormal bases (from QR of A^T and B) treated as constants in backprop, so singular values no longer scale the gradient while the forward pass still represents any rank-r update; a warm-up LoRA+SVD initialization and periodic basis refreshes complete the method.

Key results

• On LLaMA3-8B at rank 32, SDS-LoRA reaches 77.13 GSM8K and 26.36 MATH, above the best baseline (PiSSA 76.39 GSM8K, 24.80 MATH) and matching or exceeding full fine-tuning (77.04 GSM8K).
• On Gemma-2B commonsense reasoning at rank 8, SDS-LoRA averages about 73.3 vs 68.3 for the best baseline (DoRA) and 64.9 for LoRA.
• SDS-LoRA adds about 5% training time and under 1% GPU memory over LoRA on LLaMA3-8B.

Surprising findings

• SDS-LoRA at rank 32 on Gemma-2B reaches a lower final training loss than full fine-tuning, per the authors' loss curves.
• Random subspace initialization degrades performance regardless of warm-up length, and never updating Q_A and Q_B after warm-up drops average commonsense accuracy from 74.80 to 64.78.

Limitations

• Convergence analysis relies on assumptions (PL condition, negligible change in Q_A and Q_B) and idealized SGD, while experiments use AdamW.
• The deviation term E from changing bases is empirically small but lacks formal justification.
• Requires periodic QR decompositions and a warm-up stage, adding modest overhead.

Experimental setup

Datasets: Commonsense-170K MetaMathQA Code-Feedback Cars CUB200 SUN397

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-c ARC-e OBQA MATH GSM8K HumanEval Cars

Metrics: accuracy pass@1 training loss cosine similarity between full and effective gradient

Baselines: LoRA rsLoRA LoRA+ PiSSA DoRA LoRA-GA LoRA-Pro AltLoRA ScaledAdamW

Builds on: LoRA rsLoRA PiSSA LoRA+ DoRA

Closest papers by meaning

• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.94)
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.93)
• The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.93)
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.93)
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.92)
• AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.92)