Research map / Large language models / Language, safety and interpretability

Benchmarking Optimizers for Large Language Model Pretraining

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks 11 optimizers for LLM pretraining across model sizes (124M–720M, plus 520M MoE), batch sizes, and training durations with careful per-method tuning, and releases the toolkit.

Problem

Optimizer claims for LLM pretraining are validated under inconsistent protocols, making fair comparisons between methods like AdamW and newer optimizers difficult.

Approach

A controlled, large-scale benchmark that varies model size, batch size, and token budget, tunes each optimizer extensively, and revisits common setup choices (weight decay, final LR decay, z-loss, warmup, schedulers) that alter optimizer rankings.

Key results

• AdEMAMix consistently achieves state-of-the-art validation loss across batch sizes and training durations in the small and large batch settings.
• Large decoupled weight decay (0.5) helps short training horizons, while 0.1 is robust for longer training; Signum and Lion with high weight decay outperform AdamW without weight decay.
• Sophia diverges in the small-batch setting when trained beyond the Chinchilla-optimal horizon; SOAP's wall-clock slowdown grows with model size, while other optimizers have similar wall-clock time.

Surprising findings

• Sign-based methods (Signum, Lion) and MARS can outperform AdamW when trained with sufficiently large batches.
• Hyperparameter choices from popular codebases (final LR, z-loss, weight decay) can reverse the relative ranking of optimizers at Chinchilla scale.

Limitations

• Experiments limited to models up to 720M parameters and ~50B training tokens.
• Evaluates training loss only, not downstream performance.
• Several optimizers (Adan, NAdam, Shampoo, Scion) not covered.

Experimental setup

Datasets: FineWeb

Metrics: validation loss wall-clock time per 100 iterations parameter norm

Baselines: AdamW ADOPT AdEMAMix Lion Signum Muon D-Muon SOAP Sophia Schedule-Free AdamW Prodigy MARS

Builds on: AdamW Muon SOAP AdEMAMix Schedule-Free AdamW

Closest papers by meaning

• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.91)
• Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.91)
• Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures (ICLR 2025 · similarity 0.91)
• Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.91)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.91)
• A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (ICML 2025 · similarity 0.90)