AI Paper Atlas

Research map / Optimization / Optimizers

Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model

Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, Roger B Grosse

NeurIPS 2019 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 14 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add momentum to SGDSGD · Neural nets (MNIST, FMNIST, CIFAR10, LM1B), small batch sizessteps to targetNo clear effect
add momentum to SGDSGD · Neural nets, large batch sizescritical batch size / steps to targetImproved
replace SGD with Adam or K-FAC (preconditioning)SGD · Neural nets incl. Transformer on LM1Bcritical batch sizeImproved
replace SGD with Adam or K-FAC (preconditioning)SGD · Neural nets, smallest batch sizessteps to targetMixed
add momentum on top of preconditioningAdam / K-FAC · Neural netssteps to targetMixed
add exponential moving average to SGDSGD · ResNet8 CIFAR10 and Simple CNN MNIST, small batchsteps to targetImproved
add exponential moving average to SGDSGD · Neural nets, large batch sizessteps to targetNo clear effect
optimized learning rate schedule instead of fixed learning rateSGD on noisy quadratic model · Noisy quadratic modelsteps to target riskMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy