Research map / Optimization / Optimizers

What Stops SGD on LLM Pre-Training: The Need for Large Learning Rate and How to Achieve it

NeurIPS 2026 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that SGD's gap to Adam in LLM pre-training stems mainly from its inability to sustain large learning rates, and proposes ways to enable larger SGD learning rates.

Problem

SGD performs much worse than adaptive optimizers such as Adam when pre-training large language models, and the reason for this gap was unclear.

Approach

Attributes a large part of the SGD-Adam gap to SGD's inability to sustain learning rates comparable to Adam's much larger effective learning rates, supported by empirical and theoretical analysis, and offers a way to achieve large SGD learning rates.

Surprising findings

• Adam's effective learning rates are much larger than the learning rates SGD can sustain, a large part of the gap is attributed to this.

Experimental setup

Baselines: Adam

Builds on: Adam

Closest papers by meaning

• SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training (ICML 2025 · similarity 0.91)
• SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.90)
• Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.90)
• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.88)
• AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.88)
• Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)