Research map / Large language models / Instruction and fine-tuning

Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SARA, which picks low-rank optimizer subspaces by importance-sampling singular vectors of the mini-batch gradient (probability proportional to singular values) instead of taking the dominant subspace, improving LLaMA pretraining perplexity with a convergence guarantee.

Problem

Low-rank optimizers for LLM pretraining (GaLore, Fira) select the dominant gradient subspace, which freezes across training and restricts weight updates to similar subspaces, limiting the rank of cumulative updates and hurting perplexity.

Approach

Sample r of m left singular vectors of the gradient without replacement with probability proportional to their singular values, rather than taking the top-r, so adjacent subspaces differ more; this is a plug-and-play replacement for dominant subspace selection with a convergence rate comparable to GoLore.

Key results

• GaLore-SARA-Adam achieves 30.47 PPL on LLaMA-60M vs 31.50 for GaLore-Adam (full-rank Adam 27.71), and 24.21 vs 24.88 at 130M.
• SARA reduces the perplexity gap to full-rank Adam by up to 46.05% (GaLore-Adafactor, 130M).
• On LLaMA-1.1B, GaLore-SARA-Adam reaches 15.36 PPL vs 15.47 for GaLore-Adam (full-rank 15.90).

Measured effects reported in the paper

Surprising findings

• Dominant subspace overlap between adjacent update windows rises toward 1.0 in many layers after early pretraining, so the dominant subspace effectively freezes.

Limitations

• Convergence analysis relies on L-smoothness and bounded, centered mini-batch gradient noise assumptions.
• Scale-up experiments only use GaLore-Adam on LLaMA-1.1B due to limited compute.
• The convergence rate is slightly worse than GoLore since the sampling probability delta is smaller than the uniform bound r/m.

Experimental setup

Datasets: C4 SlimPajama

Benchmarks: C4 validation perplexity SlimPajama validation perplexity

Metrics: validation perplexity subspace overlap normalized singular values of weight updates

Baselines: GaLore-Adam Full-Rank Adam Fira-Adam GoLore-Adam GaLore-Adafactor GaLore-Adam-mini Online PCA (Adam)

Builds on: GaLore Fira GoLore Adam Adafactor

Closest papers by meaning

• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.91)
• A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (ICML 2025 · similarity 0.91)
• MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling (NeurIPS 2025 · similarity 0.90)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.90)
• SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training (NeurIPS 2025 · similarity 0.89)
• LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.89)