Research map / Optimization / Optimizers
Memory-efficient LLM optimizers
Designs large-batch, low-memory, and normalized optimizers for training large language models.
79 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Optimizers, keyword cluster “adaptive · optimizers · momentum”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
11 papers in 2023–24 and 41 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 4.5% to 14.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (NeurIPS 2016)
- 2017 · Neural Optimizer Search with Reinforcement Learning (ICML 2017 · 259 citations)
- 2018 · A Progressive Batching L-BFGS Method for Machine Learning (ICML 2018 · 58 citations)
- 2019 · Understanding and correcting pathologies in the training of learned optimizers (ICML 2019 · 45 citations)
- 2020 · Augment Your Batch: Improving Generalization Through Instance Repetition (CVPR 2020 · 166 citations)
- 2021 · Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling (AAAI 2021 · 14 citations)
- 2022 · Exploiting Invariance in Training Deep Neural Networks (AAAI 2022 · 2 citations)
- 2023 · Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule (ICML 2023 · 2 citations)
- 2024 · Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · 28 citations)
- 2025 · Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · 4 citations)
- 2026 · Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation (ICML 2026 · 0 citations)
Earliest papers here
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (NeurIPS 2016)
- Neural Optimizer Search with Reinforcement Learning (ICML 2017 · 259 citations)
- Learning Gradient Descent: Better Generalization and Longer Horizons (ICML 2017 · 49 citations)
What has been tried
54 papers here report 563 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 11%, hurt 24%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| update the Hessian estimate every step instead of every 10 steps | 30M GPT-2 · convergence speed per compute | 50% | Hurt | Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · 28 citations) |
| use bfloat16 weights with stochastic rounding instead of float32 weights | MATH · performance | No clear effect | Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · 4 citations) | |
| Increase Online Subspace Descent rank from 32 to 128 | C4, sequence length 1024 · perplexity | 85.90 -> 49.01 | Helped | Memory-Efficient LLM Training with Online Subspace Descent (NeurIPS 2024 · 3 citations) |
| Replace state-free SGD with signSGD | C4, LLaMA-130M, 200k iterations · validation perplexity | 19.11 -> 18.60 | Helped | FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training (ICML 2025 · 1 citation) |
| Initialize PiKE with GLaM-tuned weights instead of uniform weights | GLaM Q/A tasks · average accuracy (%) | 47.6 -> 48.1 | Helped | PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts (NeurIPS 2025 · 1 citation) |
| Use micro-batch size 1 instead of 4 | LibriSpeech test-clean · WER | 1.93 ± 0.04 -> 2.40 ± 0.13 | Hurt | Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation (ICLR 2025 · 1 citation) |
| Use full-alignment per-layer learning rates instead of global learning rates | C4 eval, 26.8B parameters · eval loss | 2.625 -> 2.576 | Helped | Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · 1 citation) |
| Increase device parameter $\tau$ | Least-squares task · asymptotic error | Helped | Towards Exact Gradient-based Training on Analog In-memory Computing (NeurIPS 2024 · 1 citation) | |
| Use GoLore (random projection) throughout instead of SVD-based projection | synthetic quadratic counterexample · convergence to optimum | Helped | Subspace Optimization for Large Language Models with Convergence Guarantees (ICML 2025 · 0 citations) | |
| Remove weight stashing in the proposed method | WikiText validation · perplexity | 27.72 -> 29.90 | Hurt | Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · 0 citations) |
| Add GradNorm | C4, LLaMA 130M · validation perplexity | Helped | SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training (ICML 2025 · 0 citations) | |
| Use MARS-approx instead of MARS-exact | GPT-2 training experiments · performance | No clear effect | MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 27 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- C4 · 8 of 27 papers
- GLUE · 4 of 27 papers
- CIFAR-10 · 4 of 27 papers
- BoolQ · 2 of 27 papers
- OpenWebText · 2 of 27 papers
- MetaMathQA · 2 of 27 papers
- SlimPajama · 2 of 27 papers
- GSM8K · 2 of 27 papers
Baselines
- AdamW · 11 of 27 papers
- Adam · 10 of 27 papers
- GaLore · 6 of 27 papers
- LoRA · 5 of 27 papers
- SGD · 5 of 27 papers
- ReLoRA · 4 of 27 papers
- Lion · 4 of 27 papers
- Muon · 4 of 27 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 52 papers; from OpenAlex and the papers' affiliations).
- Meta · 4 papers
- Google · 4 papers
- Massachusetts Institute of Technology · 2 papers
- Microsoft · 2 papers
- ByteDance · 2 papers
- Princeton University · 2 papers
- Zhejiang University · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Adam convergence and adaptive learning rates (78 papers)
- SGD momentum and step-size schedules (72 papers)
- Second-order and structured preconditioning (70 papers)
Elsewhere in the topic
- Stochastic gradient MCMC (142 papers)
- Surrogate losses and robust solvers (119 papers)
- Differentiating through embedded optimization (117 papers)
- Constrained and non-backprop training (97 papers)
- Natural gradient and Fisher approximations (77 papers)
- Randomized gradient estimation (77 papers)
- Stein variational gradient descent (47 papers)
- Gradient-based meta-learning theory (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).