Research map / Optimization / Optimizers
FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FRUGAL, a memory-efficient optimizer that splits gradients into a state-full subspace updated by Adam (AdamW) and a complementary state-free subspace updated by signSGD, with convergence guarantees for SGDM/SGD and pre-training/fine-tuning gains.
Problem
Full-parameter LLM training is memory-bound by optimizer state, and existing low-rank or blockwise memory-efficient methods (LoRA, GaLore, BAdam) update only a low-dimensional subspace per step, discarding gradient information.
Approach
Partition parameters/gradient into a state-full subspace updated with an advanced optimizer (Adam) and a complementary state-free subspace updated with stateless signSGD or SGD, so every step uses full-dimensional gradient information while optimizer state stays small; blockwise selection is used as the default projection.
Key results
- On LLaMA-130M C4 pre-training at 200k steps, FRUGAL (rho=0.25) reaches 18.60 perplexity vs 18.13 for AdamW, 21.11 for GaLore and 20.34 for BAdam under similar memory (~0.52G).
- FRUGAL with rho=0.0 (state-full only for Embeddings, RMSNorms, Output layer) outperforms GaLore and BAdam at rho=0.25 on C4 pre-training (e.g., 18.90 on 130M).
- On LLaMA 3.1-8B commonsense reasoning, FRUGAL with rank 0 reaches 84.94 average accuracy vs 84.18 for LoRA and 84.65 for GaLore (rank 32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FRUGAL instead of GaLore at ρ=0.25 | LLaMA pre-training · C4, LLaMA 60M/130M/350M/1B | validation perplexity: 25.68 -> 23.59; 21.11 -> 18.60; 16.88 -> 14.79; 13.69 -> 12.32 | Helped |
| Use FRUGAL at ρ=0.25 instead of AdamW | LLaMA pre-training · C4, LLaMA 60M/130M/350M/1B | validation perplexity: 22.73 -> 23.59; 18.13 -> 18.60; 14.43 -> 14.79; 12.02 -> 12.32 | Hurt |
| Use FRUGAL with rank 8 instead of LoRA rank 8 | RoBERTa-base fine-tuning · GLUE | Avg accuracy: 86.1 -> 86.7 | Helped |
| Replace state-free SGD with signSGD (ablation) | FRUGAL · C4, LLaMA-130M, 200k iterations | validation perplexity: 19.11 -> 18.60 | Helped |
| Move the Output layer to state-free signSGD updates (ablation) | FRUGAL, LLaMA-130M · C4, 100k iterations | perplexity: 20.02 -> 34.66 | Hurt |
| Move RMSNorms to state-free signSGD updates (ablation) | FRUGAL, LLaMA-130M · C4, 100k iterations | perplexity: 20.02 -> 20.07 | No clear effect |
| Move Embeddings to state-free signSGD updates (ablation) | FRUGAL, LLaMA-130M · C4, 100k iterations | perplexity: 20.02 -> 20.48 | Hurt |
| Set FRUGAL density to ρ=0.0 instead of ρ=0.25 (ablation) | LLaMA pre-training · C4, LLaMA 60M/130M/350M/1B | validation perplexity: 23.59 -> 24.06; 18.60 -> 18.90; 14.79 -> 15.03; 12.32 -> 12.63 | Hurt |
| Use pure bfloat16 instead of mixed precision (ablation) | FRUGAL, LLaMA-130M · C4, 100k iterations | perplexity | Hurt |
| Increase update frequency T from 10 to 200 (ablation) | FRUGAL, LLaMA-130M · C4, 200k iterations | validation perplexity: 18.82 -> 18.60 | Helped |
| Change block selection from random to ascending (ablation) | FRUGAL, LLaMA-130M · C4, 200k iterations | perplexity: 18.50 -> 18.54 | No clear effect |
| Change block selection from random to descending (ablation) | FRUGAL, LLaMA-130M · C4, 200k iterations | perplexity: 18.50 -> 18.50 | No clear effect |
Surprising findings
- Zero-density FRUGAL (signSGD on nearly all Linear layers, Adam only on Embeddings, RMSNorms and Output layer) matches or beats GaLore and BAdam run at 25% density.
- Only the Output layer is sensitive to switching from AdamW to signSGD (perplexity 20.55 to 34.66), while Embeddings and RMSNorms tolerate signSGD.
Limitations
- Convergence theory covers only SGDM as state-full and SGD as state-free optimizer with coordinate-wise projection, not the Adam/signSGD variant used in experiments.
- Output layer is highly sensitive to the state-free optimizer, so it must remain state-full, limiting the zero-state fraction.
- Concurrent-method comparisons (Fira, LDAdam, AdaMeM) are preliminary and run under different setups.
Experimental setup
Datasets: C4 GLUE Commonsense170K
Benchmarks: C4 validation perplexity GLUE commonsense reasoning (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA)
Metrics: validation perplexity accuracy Matthews/Pearson/F1 on GLUE tasks (per-task) memory estimate (GB)
Baselines: AdamW GaLore BAdam LoRA ReLoRA AdaMeM Fira LDAdam
Builds on: GaLore BAdam LoRA signSGD Adam
Closest papers by meaning
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.90)
- BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models (NeurIPS 2024 · similarity 0.89)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.89)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.89)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.87)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.