Research map / Optimization / Optimizers
Subspace Optimization for Large Language Models with Convergence Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows GaLore's SVD-based gradient projection can fail to converge to stationary points under standard assumptions, gives conditions (noise-free, large-batch, isotropic noise) for convergence, and proposes GoLore, which uses random Stiefel-manifold projections to converge with constant batch sizes at O(1/sqrt T).
Problem
GaLore, a memory-efficient subspace optimizer for LLM pre-training and fine-tuning, lacks convergence guarantees, and it is unclear whether it converges to stationary points under standard stochastic non-convex assumptions.
Approach
Identifies that GaLore's greedy SVD projection captures only gradient noise near a stationary point, so it can fail to converge; replaces the SVD projection with a uniformly random projection sampled from the Stiefel manifold (GoLore), which provably converges with constant small batches and no isotropic-noise assumption, and uses a hybrid GaLore-then-GoLore schedule in practice.
Key results
- GoLore@20% reaches 86.03 average GLUE score with RoBERTa-Base, versus 85.77 for GaLore and 86.15 for full-parameter training.
- On BoolQ fine-tuning of OPT-13B, GoLore@30% achieves 81.96 accuracy versus 79.79 for GaLore at the same 77.68 GB memory.
- Theory: GaLore's gradient norm stays bounded below by a constant on a constructed quadratic counterexample for any rank, period, and subspace optimizer, while GoLore converges at O(1/sqrt T).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GaLore's SVD projection with random Stiefel projection in the final phase (GoLore@20%) | RoBERTa-Base fine-tuning · GLUE average | Avg accuracy: +0.26 over GaLore (85.77 to 86.03) | Helped |
| Replace GaLore's SVD projection with random projection in final phase (GoLore@30%) | OPT-13B fine-tuning · BoolQ | Accuracy: +2.17 over GaLore (79.79 to 81.96) | Helped |
| Switch from GaLore to GoLore in later training phase (pre-training LLaMA-60M on C4) | LLaMA-60M pre-training · C4 | perplexity | Helped |
| Use GoLore (random projection) throughout instead of SVD-based projection (ablation) | GaLore on counterexample quadratic with stochastic noise · synthetic quadratic counterexample | convergence to optimum | Helped |
| Use large-batch gradients for the SVD step only (ablation) | GaLore on counterexample quadratic · synthetic quadratic counterexample | convergence to optimum | Helped |
| Use GaLore with small-batch stochastic gradients (SVD on noisy gradient) in the counterexample | GaLore · synthetic quadratic counterexample | convergence to optimum | Hurt |
| Use GaLore for early training phase, then GoLore for later phase (hybrid) | Practical LLM training · empirical | convergence | Helped |
| Replace GaLore/GoLore full-gradient optimizer with ReLoRA-like implementation | GaLore · computational cost analysis | computation per layer: 4bmn+4brm+6brn+5rn vs 6bmn+4rmn+2mn+3rn (theoretical) | Helped |
Surprising findings
- GaLore can fail to converge even though it is a standard descent-style method: its SVD projection can become orthogonal to the true gradient near a stationary point.
Limitations
- Convergence analysis does not yet cover the Adam optimizer.
- Does not cover recent GaLore variants such as Fira.
- Large-batch variants require batch sizes growing with T, which raises memory cost.
Experimental setup
Datasets: C4 GLUE WinoGrande BoolQ
Benchmarks: GLUE WinoGrande BoolQ C4
Metrics: perplexity accuracy memory gradient norm
Baselines: GaLore Flora Full Params.
Builds on: GaLore ReLoRA Flora
Closest papers by meaning
- A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (ICML 2025 · similarity 0.91)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.89)
- LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.88)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.88)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.86)
- From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.