Research map / Architectures and efficiency / Pruning and efficient tuning
Fantastic Pretraining Optimizers and Where to Find Them
Kaiyue Wen, David Leo Wright Hall, Tengyu Ma, Percy Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase AdamW warmup from 1000 to 2000 steps (ablation) | AdamW · 130M 1x Chinchilla C4 val loss | validation loss: 3.298 -> 3.282 | Improved |
| decrease AdamW batch size from 256 to 128 (ablation) | AdamW · 130M 1x Chinchilla C4 val loss | validation loss: 3.282 -> 3.263 | Improved |
| increase AdamW max grad norm from 1 to 2 (ablation) | AdamW · 130M 1x Chinchilla C4 val loss | validation loss: 3.263 -> 3.263 | No clear effect |
| tune the learning rate only in the GPT-3 recipe for AdamW (ablation) | AdamW · 100M model pretraining | speedup: nearly 2x speedup | Improved |
| use Muon instead of Mars | Mars · C4/EN val loss, 130M 1x Chinchilla | validation loss: 3.537 -> 3.464 | Improved |
| use Muon instead of Lion | Lion · C4/EN val loss, 520M 1x Chinchilla | validation loss: 3.108 -> 3.073 | Improved |
| use Kron or Muon instead of Mars at 8x Chinchilla | Mars · C4/EN val loss, 520M 8x Chinchilla | validation loss: 2.906 -> 2.900 | Improved |
| use Kron instead of Muon as data-to-model ratio rises to 8x | Muon · C4/EN val loss, 130M | validation loss: 3.464 -> 3.492 (1x); 3.240 -> 3.239 (8x) | Mixed |
| use Soap or NAdamW instead of Muon at 16x Chinchilla | Muon · 130M and 300M models, 16x Chinchilla | validation loss | Improved |
| use Muon instead of AdamW at 1.2B, 8x Chinchilla | AdamW · 10-task average, 1.2B | average accuracy: 67.15 -> 66.98 | No clear effect |
| use NAdamW instead of AdamW at 1.2B, 8x Chinchilla | AdamW · 10-task average, 1.2B | average accuracy: 67.15 -> 66.70 | Hurt |
| scale model from 0.1B to 1.2B for matrix-based optimizers | Muon and Soap · speedup over AdamW | speedup: 1.4x -> 1.1x | Hurt |
Closest papers by meaning
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.92)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.90)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.90)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.89)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.89)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
