AI Paper Atlas

Research map / Optimization / Optimizers

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh

ICLR 2020 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 162 citations (OpenAlex)

This paper tried a common change: Larger batch size

Increase first-stage batch size from 65536 to 131072 · BERT first training stage · speedup No clear effect

What 785 accepted papers report for this change

See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace the baseline optimizer with LAMBBERT-Large · SQuAD-v1 devF1: 90.395 -> 91.752Improved
Scale BERT training to batch 32k with LAMBBERT-Large · SQuAD-v1 devF1: 90.395 -> 91.475Improved
Use mixed-batch training at 64k/32kBERT-Large · SQuAD-v1 devF1: 90.395 -> 90.584Improved
Use LAMB instead of LARSBERT-Large · SQuAD-v1 dev, batch 32kF1Improved
Use the warmup and decay learning-rate schemeAdagrad on ResNet-50 · ImageNet validation, batch 16Ktop-1 accuracy: 0.5538 -> 0.7201Improved
Use the warmup and decay learning-rate schemeAdam on ResNet-50 · ImageNet validation, batch 16Ktop-1 accuracy: 0.6604 -> 0.7348Improved
Use the warmup and decay learning-rate schemeAdamW on ResNet-50 · ImageNet validation, batch 16Ktop-1 accuracy: 0.6727 -> 0.7307Improved
Use LAMB instead of momentumResNet-50 · ImageNet validation, batch 16Ktop-1 accuracy: 0.7520 -> 0.7666Improved
Replace momentum SGD with LAMBDavidNet · CIFAR-10 test, batch 512test accuracy: 0.9372 -> 0.9408Improved
Replace AdamW with LAMBLeNet · MNIST test, batch 1024average accuracy over 5 runs: 0.9941 -> 0.9945Improved
Remove Adam bias correction from LAMB (ablation)LAMB · BERT and ImageNet trainingtest or validation accuracyNo clear effect
Replace the L2 norm with other layer norms (ablation)LAMB · ImageNet validation, ResNet-50validation accuracy: less than 0.1 percentNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy