Research map / Architectures and efficiency / Pruning and efficient tuning
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaLore, which projects weight gradients into a low-rank subspace (via periodic SVD) so Adam-style optimizer states are stored in compact form, enabling full-parameter LLM pre-training and fine-tuning with much lower memory.
Problem
Memory cost of weights, gradients and optimizer states in LLM pre-training and fine-tuning; low-rank adaptation methods like LoRA underperform full-rank training and can require full-rank warmup.
Approach
Instead of low-rank parameterizing the weights (LoRA/ReLoRA), project the gradient matrix into a low-rank subspace using SVD-derived projectors refreshed every T steps, run the optimizer on the compact projected gradient, and project the update back, so full-parameter learning is preserved while optimizer states shrink.
Key results
- On LLaMA 1B pre-training on C4, GaLore reaches validation perplexity 15.64 vs 15.56 for full-rank, with optimizer memory 4.38G vs 7.80G.
- 8-bit GaLore pre-trains LLaMA 7B to perplexity 14.65 vs 14.61 for 8-bit Adam, with total memory 22.0G (fits RTX 4090 24GB) and 63.3% total memory reduction vs BF16 Adam.
- Fine-tuning RoBERTa-Base on GLUE at rank 4, GaLore averages 85.89 vs LoRA 85.61 with 253M vs 257M memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GaLore instead of full-rank Adam training | LLaMA pre-training · C4 validation, 60M–1B models | perplexity: 34.06 -> 34.88; 25.08 -> 25.36; 18.80 -> 18.95; 15.56 -> 15.64 | Hurt |
| Use GaLore instead of LoRA | LLaMA pre-training · C4 validation, 60M–1B models | perplexity: 34.99 -> 34.88; 33.92 -> 25.36; 25.58 -> 18.95; 19.21 -> 15.64 | Helped |
| Use GaLore instead of ReLoRA | LLaMA pre-training · C4 validation, 60M–1B models | perplexity: 37.04 -> 34.88; 29.37 -> 25.36; 29.08 -> 18.95; 18.33 -> 15.64 | Mixed |
| Replace 8-bit Adam with 8-bit GaLore | LLaMA 7B pre-training · C4, 40K–150K steps | perplexity: 18.09 -> 17.94; 15.47 -> 15.39; 14.83 -> 14.95; 14.61 -> 14.65 | Mixed |
| Use GaLore rank 4 instead of LoRA rank 4 | RoBERTa-Base fine-tuning · GLUE average | average score: 85.61 -> 85.89 | Helped |
| Use GaLore rank 8 instead of LoRA rank 8 | RoBERTa-Base fine-tuning · GLUE average | average score: 85.93 -> 85.94 | No clear effect |
| Change subspace-switching frequency to be too frequent or too slow (ablation) | GaLore pre-training · C4 | convergence | Hurt |
| Reduce subspace rank (ablation) | GaLore pre-training · C4 | convergence rate | Hurt |
| Train rank 128 for 80K steps instead of rank 512 for 20K steps (ablation) | GaLore · C4 pre-training | loss: rank of 128 using 80K steps; rank of 512 using 20K steps | Helped |
Surprising findings
- GaLore with rank 1024 on 1B model slightly outperforms the full-rank baseline in perplexity (15.56 vs. 15.64 comparison noted by authors for rank 1024).
- Pure low-rank weight factorization (Low-Rank) performs far worse than LoRA and GaLore at 1B (142.53 perplexity).
Limitations
- Throughput overhead of about 17% vs 8-bit Adam in the current implementation.
- 7B pre-training compared to 8-bit Adam only with a single untuned trial due to compute constraints.
- Projection matrices themselves are not memory-optimized and are left for future work.
Experimental setup
Datasets: C4 GLUE
Benchmarks: C4 validation perplexity GLUE
Metrics: validation perplexity GLUE average score memory estimate (GB) throughput (tokens/s)
Baselines: Full-Rank Low-Rank LoRA ReLoRA 8-bit Adam Adafactor AdamW
Builds on: LoRA ReLoRA Adam 8-bit optimizers Adafactor
Closest papers by meaning
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.93)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.92)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.92)
- Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models (ICLR 2025 · similarity 0.92)
- VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections (NeurIPS 2024 · similarity 0.91)
- FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.