Research map / Optimization / Optimizers

SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training

NeurIPS 2025 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SUMO, a memory-efficient low-rank optimizer that performs exact SVD-based moment orthogonalization inside a dynamically updated low-rank subspace, claiming faster convergence and up to 20% lower memory than GaLore-style methods.

Problem

Low-rank gradient optimizers for LLM training focus on memory savings but ignore convergence acceleration; Newton-Schulz approximations to moment orthogonalization (Muon) incur errors that grow with the moment's condition number.

Approach

Perform exact SVD orthogonalization of the first-order moment within a randomized-SVD-derived low-rank subspace (refreshed every K steps), giving norm-induced steepest descent with only one moment state, plus a norm-growth limiter and a residual-gradient update in the original space.

Key results

• On GLUE with RoBERTa-base at rank 4, SUMO (SVD) reaches 197M memory versus GaLore's 253M and averages higher accuracy on most tasks (e.g., RTE 81.07 vs 79.42).
• On C4 pre-training, SUMO achieves validation perplexity 14.68 at 1B with 3.84G memory versus GaLore's 15.64 at 4.38G.
• On GSM8K with LLaMA-3B (8-shot), SUMO reaches 76.7% versus GaLore's 74.9%.

Measured effects reported in the paper

Surprising findings

• Reported that in their experiments SUMO matches or beats full-rank training on several metrics, e.g., C4 1B perplexity 14.68 vs 15.56 for full-rank.

Limitations

• Convergence and error bounds depend on an assumed bound delta on Newton-Schulz error and the theory is only valid for delta < 1/(4 sqrt(n)).
• Relies on empirically observed low-rank, ill-conditioned moment structure; the rank-decay lemma depends on reversible network assumptions.
• SVD cost is higher than Newton-Schulz for some shapes (paper's own FLOP example shows roughly twice the operations).

Experimental setup

Datasets: C4 GLUE GSM8K

Benchmarks: GLUE C4 validation perplexity GSM8K BoolQ RTE HellaSwag WinoGrande OpenBookQA ARC-E ARC-C PIQA SciQ

Metrics: accuracy Matthews correlation Pearson correlation F1 validation perplexity memory (GB)

Baselines: GaLore LoRA ReLoRA Adam APOLLO APOLLO-Mini AdamW Full fine-tuning Newton-Schulz5 (SUMO variant)

Builds on: GaLore Muon AdaRankGrad OSGDM

Closest papers by meaning

• Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.89)
• A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (ICML 2025 · similarity 0.89)
• Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives (ICLR 2025 · similarity 0.88)
• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.88)
• AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.88)
• Memory-Efficient LLM Training with Online Subspace Descent (NeurIPS 2024 · similarity 0.88)