Research map / Optimization / Optimizers
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, Roger B Grosse
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add momentum to SGD | SGD · Neural nets (MNIST, FMNIST, CIFAR10, LM1B), small batch sizes | steps to target | No clear effect |
| add momentum to SGD | SGD · Neural nets, large batch sizes | critical batch size / steps to target | Improved |
| replace SGD with Adam or K-FAC (preconditioning) | SGD · Neural nets incl. Transformer on LM1B | critical batch size | Improved |
| replace SGD with Adam or K-FAC (preconditioning) | SGD · Neural nets, smallest batch sizes | steps to target | Mixed |
| add momentum on top of preconditioning | Adam / K-FAC · Neural nets | steps to target | Mixed |
| add exponential moving average to SGD | SGD · ResNet8 CIFAR10 and Simple CNN MNIST, small batch | steps to target | Improved |
| add exponential moving average to SGD | SGD · Neural nets, large batch sizes | steps to target | No clear effect |
| optimized learning rate schedule instead of fixed learning rate | SGD on noisy quadratic model · Noisy quadratic model | steps to target risk | Mixed |
Closest papers by meaning
- How Does Critical Batch Size Scale in Pre-training? (ICLR 2025 · similarity 0.89)
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training (NeurIPS 2025 · similarity 0.88)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.87)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.87)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.86)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
