Research map / Architectures and efficiency / Pruning and efficient tuning
Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fira, a plug-and-play low-rank training framework that achieves full-rank gradient training under a low-rank constraint via norm-based scaling of the residual gradient and a norm-growth limiter against loss spikes.
Problem
Low-rank training methods for LLMs (LoRA, GaLore) restrict optimization to a low-rank subspace, discarding gradient information and yielding suboptimal performance versus full-rank training, while still needing to save optimizer memory.
Approach
Observes that the scaling factor (norm ratio of Adam-corrected to raw gradient) is similar between low-rank and full-rank training, so the low-rank Adam scaling is reused to correct the out-of-subspace residual gradient (norm-based scaling), and a norm-growth limiter caps sudden gradient-norm increases to prevent loss spikes.
Key results
- Pre-training LLaMA 60M on C4 reaches 31.06 perplexity vs 34.06 for full-rank and 34.88 for GaLore at the same rank setting.
- For LLaMA 1B, Fira reaches 14.31 perplexity vs 15.56 full-rank and 15.64 GaLore, while using 6.98G optimizer-related memory (vs 10.40G full-rank).
- Fine-tuning LLaMA 7B on eight commonsense tasks gives 76.9 average accuracy vs 74.7 LoRA, 62.7 GaLore, and 58.6 full-rank.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add norm-based scaling to correct gradients outside the low-rank subspace (ablation) | Fira · C4, LLaMA 60M | validation perplexity: 37.06 -> 31.06 | Helped |
| use column-level rather than matrix-level scaling factors (ablation) | Fira · C4, LLaMA 60M | validation perplexity: 31.52 -> 31.06 | Helped |
| add the norm-growth limiter (ablation) | Fira · C4, LLaMA 60M | validation perplexity: 32.22 -> 31.06 | Helped |
| replace the norm-growth limiter with gradient clipping (ablation) | Fira · C4, LLaMA 60M | validation perplexity: 31.06 -> 31.22 | Hurt |
| replace the norm-growth limiter with gradient shrink (ablation) | Fira · C4, LLaMA 60M | validation perplexity: 31.06 -> 33.98 | Hurt |
| replace the norm-growth limiter with tensor-wise scaling (ablation) | Fira · C4, LLaMA 60M | validation perplexity: 31.06 -> 33.81 | Hurt |
| use column-level instead of matrix-level scaling (ablation) | Fira-only-scaling · C4, LLaMA 60M | validation perplexity: 32.05 -> 31.68 | Helped |
| use SVD for low-rank gradient sampling (ablation) | Fira-only-scaling · C4, LLaMA 60M | validation perplexity: 32.22 -> 31.68 | Helped |
| increase rank from 4 to 128 (ablation) | Fira-only-scaling · C4, LLaMA 60M | validation perplexity: 35.91 -> 31.68 | Helped |
| train with Fira instead of GaLore | LLaMA pre-training · C4, LLaMA 60M–1B | validation perplexity: 34.88 -> 31.06; 25.36 -> 22.73; 18.95 -> 16.85; 15.64 -> 14.31 | Helped |
| train with Fira instead of full-rank training | LLaMA pre-training · C4, LLaMA 60M–1B | validation perplexity: 34.06 -> 31.06; 25.08 -> 22.73; 18.80 -> 16.85; 15.56 -> 14.31 | Helped |
| fine-tune with Fira instead of the strongest baseline | LLaMA 7B · Commonsense reasoning average | accuracy: 74.7 -> 76.9 | Helped |
Surprising findings
- Fira's pre-training perplexity surpasses full-rank Adam, which the authors attribute to extra randomness in the gradient direction helping escape local optima.
- GaLore-add (adding the residual gradient with plain SGD) gives almost no improvement over GaLore, indicating that uncorrected residual gradients are ineffective.
Limitations
- Theoretical analysis relies on strong assumptions (bounded scaling factors, Gaussian gradients in early training) and is approximate.
- Baseline results for most methods are taken from prior work rather than re-run under identical conditions, and the 7B pre-training comparison is only against GaLore.
- Real throughput of Fira is somewhat lower than GaLore and other memory-efficient methods (e.g., 94.2% vs 95.9% normalized throughput for LLaMA 1B).
Experimental setup
Datasets: C4 Commonsense reasoning benchmark (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA)
Benchmarks: C4 validation perplexity BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: validation perplexity accuracy memory estimate normalized throughput
Baselines: GaLore LoRA ReLoRA Flora Full-rank training GaLore-add gradient clipping tensor-wise scaling
Builds on: GaLore LoRA ReLoRA Adam
Closest papers by meaning
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.93)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.92)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.91)
- VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections (NeurIPS 2024 · similarity 0.90)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.90)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.