Research map / Architectures and efficiency / Pruning and efficient tuning
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA-RITE, a transformation-invariant adaptive matrix-preconditioned optimizer for LoRA that preconditions only the low-rank side at near-Adam cost, outperforming Adam, LoRA+, ScaledAdam, Shampoo and Lamb on Gemma and mT5 finetuning.
Problem
Standard optimizers (Adam, Adagrad, Shampoo, etc.) applied to LoRA factors A and B are not invariant to how the same weight update Z = AB^T is factorized, leading to imbalanced factor updates and suboptimal finetuning.
Approach
Use unmagnified gradients (gradients multiplied by the orthogonal bases of the other factor, removing magnitude dependence) with one-sided matrix preconditioning on the r-by-r side, plus basis-transport of accumulated first and second moments and an escaped-mass correction, yielding a provably transformation-invariant update at Adam-like cost.
Key results
- On Gemma-2B, LoRA-RITE reaches 58.81 average on four LLM benchmarks vs 54.33 for Adam, with GSM8K 30.10 vs 24.26.
- On Gemma-7B GSM8K, LoRA-RITE gets 55.50% vs Adam 48.37% and Lamb 50.64%.
- LoRA-RITE runs about 8% slower than Adam on Gemma-2B and 5% slower on Gemma-7B in steps per second.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Adam with LoRA-RITE | LoRA fine-tuning of Gemma-2B · Super-Natural Instructions global validation | exact match accuracy: 50.51 -> 55.11 | Helped |
| replace Lamb with LoRA-RITE | LoRA fine-tuning of Gemma-7B · GSM8K test | accuracy: 50.64 -> 55.50 | Helped |
| replace Lamb with LoRA-RITE | LoRA fine-tuning of Gemma-2B · GSM8K test | accuracy: 26.76 -> 30.10 | Helped |
| replace Lamb with LoRA-RITE | LoRA fine-tuning of Gemma-2B · HellaSwag | accuracy: 86.60 -> 87.28 | Helped |
| replace Lamb with LoRA-RITE | LoRA fine-tuning of Gemma-7B · OpenBookQA dev | accuracy: 83.20 -> 84.80 | Helped |
| use LoRA rank 4 instead of rank 16 (ablation) | LoRA-RITE on Gemma-2B · OpenBookQA dev | accuracy: 68.8 -> 70.4 | Helped |
| use LoRA rank 64 instead of rank 16 (ablation) | LoRA-RITE on Gemma-2B · GSM8K test | accuracy: 30.10 -> 31.46 | Helped |
| use LoRA rank 64 instead of rank 16 (ablation) | LoRA-RITE on Gemma-2B · OpenBookQA dev | accuracy: 68.8 -> 68.8 | No clear effect |
| use LoRA rank 16 instead of rank 4 (ablation) | LoRA-RITE on mT5-XXL · OpenBookQA dev | accuracy: 74.80 -> 75.00 | No clear effect |
| replace Adam with LoRA-RITE | LoRA fine-tuning of Gemma-2B · OpenBookQA dev training speed | steps per second: 0.948 -> 0.878 | Hurt |
Surprising findings
- Lamb, which enforces scalar scale invariance but not full transformation invariance, is consistently the second-best optimizer, suggesting transformation invariance matters.
- Diagonal preconditioners cannot achieve transformation invariance for LoRA, so matrix preconditioning is needed.
Limitations
- Theoretical convergence guarantee depends on an additional smoothness assumption on the basis and magnitude changes.
- Focuses on LoRA of form f(AB^T); does not cover other parameter-efficient structures or more complex LoRA variants.
- Gains are reported on a limited set of tasks and models, with learning rates tuned per optimizer.
Experimental setup
Datasets: Super-Natural Instructions HellaSwag ArcChallenge GSM8K OpenBookQA
Benchmarks: Super-Natural Instructions HellaSwag ArcChallenge GSM8K OpenBookQA
Metrics: exact match accuracy ROUGE-L training steps per second
Baselines: Adam LoRA+ ScaledAdam Shampoo Lamb
Builds on: LoRA Adagrad Shampoo LoRA+ ScaledAdam
Closest papers by meaning
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.90)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.89)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.89)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.89)
- Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models (ICML 2024 · similarity 0.89)
- Activated LoRA: Fine-tuned LLMs for Intrinsics (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.