Research map / Large language models / Instruction and fine-tuning
A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Randomized Subspace Optimization (RSO), which trains LLMs by randomly projecting weights into low-dimensional subspaces and solving each subproblem with Adam, reducing both optimizer-state and activation memory, with convergence guarantees.
Problem
Memory for LLM training is dominated by optimizer states and activations; existing memory-efficient methods like GaLore reduce only optimizer state and lack thorough convergence theory for periodically updated projections.
Approach
Decompose full-parameter training into a sequence of random low-dimensional subproblems: at each iteration sample random projection matrices P, optimize small matrices B with an inexact solver (e.g., Adam), and update W += P B. Because only projected inputs (XP) and reduced gradients need storage, activation memory also shrinks, unlike GaLore which needs full-rank activations.
Key results
- On LLaMA-1B pre-training on C4, RSO reaches 15.86 perplexity versus 15.56 for Adam, with optimizer-state memory reduced by about 70.7% relative to Adam (1.46G vs 4.99G).
- On GLUE fine-tuning of RoBERTa-Base with rank 8, RSO averages 86.77 versus 86.28 for Adam and 85.94 for GaLore.
- For LLaMA-7B with sequence length 64, RSO's iteration time is 2.40 s versus 7.86 s for GaLore and 7.84 s for Adam, and peak memory is 54.81 GB versus 75.33 GB for GaLore.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RSO instead of Adam | LLaMA pre-training · C4, LLaMA-350M | validation perplexity: 18.80 -> 18.86 | No clear effect |
| Use RSO instead of GaLore | LLaMA pre-training · C4, LLaMA-7B, 50K steps | training time (h): 134 -> 64 | Helped |
| Use RSO instead of Adam | LLaMA-7B pre-training · C4, LLaMA-7B, 50K steps | memory (GB): 78.92 -> 54.81 | Helped |
| Use RSO instead of Adam | RoBERTa-Base fine-tuning · GLUE, rank 8 | average score: 86.28 -> 86.77 | Helped |
| Use RSO instead of GaLore | LLaMA-7B fine-tuning · COPA | test accuracy: 85.0 -> 86.0 | Helped |
| Use RSO instead of LoRA | LLaMA-13B fine-tuning · WinoGrande | test accuracy: 76.6 -> 74.7 | Hurt |
| Use RSO instead of GaLore | OPT-6.7B fine-tuning · WinoGrande | test accuracy: 66.8 -> 69.2 | Helped |
| Use RSO instead of GaLore | OPT-6.7B fine-tuning · COPA | test accuracy: 80.0 -> 82.0 | Helped |
Surprising findings
- RSO with Adam-solved subproblems slightly outperforms Adam on several GLUE tasks (e.g., CoLA and MRPC at rank 8), which the authors attribute to the subspace method rather than to a loss in quality.
Limitations
- Final pre-training of LLaMA-7B was only run for 50K of 150K steps due to limited compute.
- Perplexity on LLaMA-7B at 50K steps is slightly worse than Adam and GaLore (15.99 vs 15.43 and 15.59).
- Convergence theory gives O(epsilon^-1) iterations for stationarity with inexact subproblem solutions and relies on a specific random projection distribution assumption (P^T P scaled identity), which Gaussian sampling only approximates.
Experimental setup
Datasets: C4 GLUE WinoGrande COPA
Benchmarks: C4 GLUE WinoGrande COPA
Metrics: validation perplexity accuracy peak memory (GB) iteration time (s)
Baselines: Adam GaLore LoRA ReLoRA
Builds on: GaLore LoRA ReLoRA Adam random coordinate descent
Closest papers by meaning
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.91)
- Subspace Optimization for Large Language Models with Convergence Guarantees (ICML 2025 · similarity 0.91)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.90)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.90)
- Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures (ICLR 2025 · similarity 0.90)
- LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.