Research map / Optimization / Optimizers
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh
This paper tried a common change: Larger batch size
Increase first-stage batch size from 65536 to 131072 · BERT first training stage · speedup No clear effect
What 785 accepted papers report for this change
- 469 improved
- 107 no clear effect
- 48 mixed
- 161 hurt
- Predicting the Emergence of Induction Heads in Language Model Pretraining ICML 2026 · increase batch size (fixed context 1024) · Natural data (CC100) · induction head emergence point (updates): rho=-1.0, p<0.001Improved
- When RL Meets Adaptive Speculative Training: A Unified Training-Serving System ICML 2026 · Increase batch size from 1 to 32 · MiniMax M2.1 FP8 · speedup: 1.57x -> 1.25xHurt
- Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking ICML 2026 · increase per-device batch size to 10 using freed memory · Llama 3-8B, 128K · throughput (TPS): 2464.10 (Ulysses) -> 2526.77 (UPipe)Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the baseline optimizer with LAMB | BERT-Large · SQuAD-v1 dev | F1: 90.395 -> 91.752 | Improved |
| Scale BERT training to batch 32k with LAMB | BERT-Large · SQuAD-v1 dev | F1: 90.395 -> 91.475 | Improved |
| Use mixed-batch training at 64k/32k | BERT-Large · SQuAD-v1 dev | F1: 90.395 -> 90.584 | Improved |
| Use LAMB instead of LARS | BERT-Large · SQuAD-v1 dev, batch 32k | F1 | Improved |
| Use the warmup and decay learning-rate scheme | Adagrad on ResNet-50 · ImageNet validation, batch 16K | top-1 accuracy: 0.5538 -> 0.7201 | Improved |
| Use the warmup and decay learning-rate scheme | Adam on ResNet-50 · ImageNet validation, batch 16K | top-1 accuracy: 0.6604 -> 0.7348 | Improved |
| Use the warmup and decay learning-rate scheme | AdamW on ResNet-50 · ImageNet validation, batch 16K | top-1 accuracy: 0.6727 -> 0.7307 | Improved |
| Use LAMB instead of momentum | ResNet-50 · ImageNet validation, batch 16K | top-1 accuracy: 0.7520 -> 0.7666 | Improved |
| Replace momentum SGD with LAMB | DavidNet · CIFAR-10 test, batch 512 | test accuracy: 0.9372 -> 0.9408 | Improved |
| Replace AdamW with LAMB | LeNet · MNIST test, batch 1024 | average accuracy over 5 runs: 0.9941 -> 0.9945 | Improved |
| Remove Adam bias correction from LAMB (ablation) | LAMB · BERT and ImageNet training | test or validation accuracy | No clear effect |
| Replace the L2 norm with other layer norms (ablation) | LAMB · ImageNet validation, ResNet-50 | validation accuracy: less than 0.1 percent | No clear effect |
Closest papers by meaning
- MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training (ICML 2025 · similarity 0.88)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.87)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.86)
- LEMON: Lossless model expansion (ICLR 2024 · similarity 0.86)
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.86)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
