Research map / Architectures and efficiency / Pruning and efficient tuning

AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaPreLoRA, a LoRA optimizer that pairs an Adafactor diagonal-Kronecker preconditioner on the full weight space with a closed-form, H_t-balanced selection from the affine solution set of the singular factor-space system.

Problem

LoRA optimization in factor space: the Jacobian of the factor-to-weight generator is rank-deficient, so the factor-space preconditioner induced by a W-space preconditioner is singular and the factor update is non-unique.

Approach

Unifies LoRA optimizers as choices of a W-space preconditioner and a rule selecting an element of the affine solution set; uses the Adafactor diagonal Kronecker preconditioner H_t and selects the factor update minimizing the H_t-weighted imbalance between the two factor contributions, giving a closed-form update at O((m+n)r) memory.

Key results

• On GPT-2 small E2E at r=4, AdaPreLoRA AdamW reaches BLEU 70.0 vs 69.5 for Scaled AdamW and 69.2 for LoRA-Pro AdamW.
• On Qwen2-7B, AdaPreLoRA AdamW gets 85.6 on ARC and 76.4 on GSM8K, best among compared methods.
• On Mistral-7B RTE, AdaPreLoRA AdamW uses 26.0 GB peak memory versus 50.4 GB for LoRA-Pro AdamW.

Surprising findings

• Gradient-statistics-aware baselines with O(mn) memory (LoRA-Pro AdamW, SOAP) trail vanilla AdamW on several Qwen2-7B tasks.

Limitations

• Per-step time is substantially higher than vanilla AdamW (0.99 vs 0.24 s/step on Mistral-7B).
• Extensions to MoE, quantized backbones (QLoRA) and diffusion transformers are left to future work.
• Learning rates are tuned per optimizer by grid search, so comparisons depend on tuning effort.

Experimental setup

Datasets: E2E DART GLUE (RTE, CoLA, MRPC) ARC GSM8K Mix-of-Show

Benchmarks: E2E DART GLUE ARC GSM8K Mix-of-Show

Metrics: BLEU NIST METEOR ROUGE-L CIDEr accuracy CLIP score FID peak GPU memory s/step

Baselines: AdamW (LoRA) SGD (LoRA) Scaled GD Scaled AdamW LoRA-Pro SGD LoRA-Pro AdamW SOAP

Builds on: LoRA Adafactor LoRA-Pro Scaled GD / Riemannian Preconditioned LoRA Imbalance-Regularized LoRA

Closest papers by meaning

• Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models (ICML 2024 · similarity 0.91)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization (ICLR 2025 · similarity 0.88)
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.88)
• A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models (ICML 2025 · similarity 0.88)
• Flora: Low-Rank Adapters Are Secretly Gradient Compressors (ICML 2024 · similarity 0.88)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.88)