Research map / Optimization / Implicit bias and generalization
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use large batches instead of batch size 256 | ADAM · MNIST, TIMIT, CIFAR-10, and CIFAR-100 | testing accuracy: 98.03% ± 0.07% -> 97.81% ± 0.07%; 64.02% ± 0.2% -> 59.45% ± 1.05%; 80.04% ± 0.12% -> 77.26% ± 0.42%; 89.24% ± 0.12% -> 8 | Hurt |
| Use large-batch rather than small-batch training | ADAM · F1, F2, C1, C2, C3, and C4 sharpness | sharpness (full space, epsilon=10^{-3}): 1.23 ± 0.83 -> 205.14 ± 69.52; 1.39 ± 0.02 -> 310.64 ± 38.46; 28.58 ± 3.13 -> 707.23 ± 43.04; 8.68 ± 1.32 -> 925.32 ± 38 | Hurt |
| Add data augmentation (ablation) | Large-batch ADAM · CIFAR-10 and CIFAR-100 | testing accuracy: C1: 80.04% ± 0.12% -> 82.50% ± 0.67%; C2: 89.24% ± 0.12% -> 90.26% ± 1.15%; C3: 49.58% ± 0.39% -> 53.03% ± 0.33%; C4: 63 | Mixed |
| Add conservative proximal training (ablation) | Large-batch ADAM · MNIST, TIMIT, CIFAR-10, and CIFAR-100 | testing accuracy: F1: 98.03% ± 0.07% -> 98.12% ± 0.01%; F2: 64.02% ± 0.2% -> 61.94% ± 1.10%; C1: 80.04% ± 0.12% -> 78.41% ± 0.22%; C2: 89. | Mixed |
| Add adversarial training (ablation) | Large-batch training · Experiments in the paper | testing accuracy | No clear effect |
| Add stability training (ablation) | Large-batch training · Experiments in the paper | testing accuracy | No clear effect |
| Warm-start large-batch training with sufficiently many small-batch epochs (ablation) | ADAM on F2 and C1 · TIMIT and CIFAR-10 | testing accuracy | Helped |
| Warm-start large-batch training with only a few small-batch epochs (ablation) | ADAM on F2 and C1 · TIMIT and CIFAR-10 | testing accuracy | No clear effect |
| Increase batch size beyond the observed threshold (ablation) | ADAM on F2 and C1 · TIMIT and CIFAR-10 | testing accuracy | Hurt |
Closest papers by meaning
- A Bayesian Perspective on Generalization and Stochastic Gradient Descent (ICLR 2018 · similarity 0.93)
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (ICML 2020 · similarity 0.93)
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks (NeurIPS 2017 · similarity 0.93)
- Extrapolation for Large-batch Training in Deep Learning (ICML 2020 · similarity 0.93)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.93)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.