Research map / Architectures and efficiency / Pruning and efficient tuning
Flora: Low-Rank Adapters Are Secretly Gradient Compressors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LoRA updates approximate a fixed random projection of gradients, and proposes Flora, which resamples random down/up projections to get high-rank updates with sublinear memory for gradient accumulation and momentum.
Problem
Optimizer state memory for training large neural networks grows linearly with model size, and LoRA reduces it only by restricting weight updates to low rank, hurting performance.
Approach
LoRA's update is dominated by a random projection of the gradient (B-only LoRA ≈ random projection). Flora compresses gradient accumulation and momentum by random down-projection and resamples the projection matrix over time, so the total weight change is not restricted to a low-rank subspace while optimizer states stay O(nr).
Key results
- On T5-3B XSum with gradient accumulation, Flora(256) reaches 44.3/20.7/36.2 ROUGE vs 42.5/19.1/34.6 without accumulation, with 2.8 GiB overhead vs 9.9 GiB for naive accumulation.
- On GPT-2 110M IWSLT17 momentum, Flora(256) achieves 19.9 BLEU matching no-momentum baseline 19.4 and naive 19.9, while LoRA(256) reaches only 9.83.
- On C4 pretraining, Flora gives lower perplexity than GaLore (32.52 vs 34.64 at 60M; 23.69 vs 27.17 at 350M) at similar memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Flora rank 512 instead of naive gradient accumulation | T5-3B · XSum | ROUGE-1/ROUGE-2/ROUGE-L: 44.4/20.9/36.3 -> 44.5/20.9/36.4 | No clear effect |
| Use Flora rank 512 instead of naive gradient accumulation | GPT-2-XL · IWSLT17 De-En | BLEU: 33.2 -> 33.6 | Helped |
| Use Flora rank 512 instead of naive gradient accumulation | GPT-2-XL · IWSLT17 De-En | peak memory (GiB): 26.5 -> 22.8 | Helped |
| Use Flora rank 256 instead of LoRA rank 256 | GPT-2 base · IWSLT17 De-En gradient accumulation | BLEU: 13.4 -> 25.4 | Helped |
| Use Flora rank 256 instead of LoRA rank 256 | T5-small · XSum momentum | ROUGE-1/ROUGE-2/ROUGE-L: 22.2/5.04/17.9 -> 30.2/9.51/24.0 | Helped |
| Use Flora instead of Adam | ViT Base · CIFAR-100 | accuracy: 91.93 -> 92.15 | Helped |
| Use Flora instead of Adam | ViT Large · CIFAR-100 | accuracy: 92.97 -> 92.98 | No clear effect |
| Use Flora instead of GaLore | 60M language model · C4 | perplexity: 34.64 -> 32.52 | Helped |
| Use Flora instead of GaLore | 350M language model · C4 | perplexity: 27.17 -> 23.69 | Helped |
| Use Flora rank 256 instead of naive accumulation | T5-small · XSum with linear-memory optimizer | ROUGE-1/ROUGE-2/ROUGE-L: 34.0/11.5/26.7 -> 33.5/11.1/26.3 | Hurt |
| Set momentum resampling interval κ to 1 rather than 1000 (ablation) | Flora on T5-small · XSum momentum | ROUGE-1/ROUGE-2/ROUGE-L: 30.4/9.70/24.2 -> 0.00/0.00/0.00 | Hurt |
| Set momentum resampling interval κ to 100 (ablation) | Flora on T5-small · XSum momentum | ROUGE-1/ROUGE-2/ROUGE-L: κ=100: 29.3/8.89/23.2; κ=1000: 30.4/9.70/24.2 | Mixed |
Surprising findings
- LoRA increases memory usage in gradient accumulation with Adafactor because Adafactor's state is already sublinear and LoRA adds low-rank matrices and an extra Jacobian path.
- Resampling the down-projection matrix is what restores performance; fixed-projection random projection (RP) and LoRA(B) perform much worse than resampled RP (RRP).
Limitations
- Neither LoRA nor Flora saves memory for back-propagation; the full gradient is still computed per layer.
- Momentum compression requires resampling interval kappa tuning; too-large kappa degrades performance due to rank bottleneck within each interval.
- Largest experiments are at 7B only in a limited C4/memory setting; GPT-3-scale results are estimated, not empirically verified.
Experimental setup
Datasets: XSum IWSLT17 German-English CIFAR-100 Fashion-MNIST C4
Benchmarks: XSum IWSLT17 De-En CIFAR-100 C4
Metrics: ROUGE-1/2/L SacreBLEU accuracy peak memory (GiB) perplexity
Baselines: LoRA Naive accumulation/momentum Adafactor GaLore Adam ReLoRA
Builds on: LoRA Adafactor Johnson-Lindenstrauss lemma GaLore
Closest papers by meaning
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.90)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.90)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.89)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.89)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.89)
- A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.