Research map / Optimization / Implicit bias and generalization
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale the learning rate with the square root of batch size (ablation) | large-batch training · CIFAR-10 (Resnet44) | validation accuracy: 86.10% -> 89.30% | Helped |
| add Ghost Batch Normalization using virtual batches (ablation) | large-batch training with tuned learning rate · CIFAR-10 (Resnet44) | validation accuracy: 89.30% -> 90.50% | Helped |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · CIFAR-10 (Resnet44) | validation accuracy: 90.50% -> 93.07% | Helped |
| use large batch rather than small batch without adapting training | Resnet44 · CIFAR-10 | validation accuracy: 92.83% -> 86.10% | Hurt |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · MNIST (F1) | validation accuracy: 97.60% -> 98.53% | Helped |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · CIFAR-10 (C1) | validation accuracy: 86.4% -> 88.20% | Helped |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · CIFAR-10 (VGG) | validation accuracy: 91.50% -> 93.03% | Helped |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · CIFAR-100 (C3) | validation accuracy: 57.5% -> 63.20% | Helped |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · CIFAR-100 (WResnet16-4) | validation accuracy: 71.20% -> 73.57% | Helped |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · ImageNet (Alexnet, LB size 4096) | top-1 accuracy: 54.92% -> 59.5% | Helped |
| adapt the training regime to match the small-batch update count (ablation) | large-batch training with tuned learning rate and GBN · ImageNet (Alexnet, LB size 8192) | top-1 accuracy: 53.93% -> 59.5% | Helped |
| use dropout, dropconnect, or label noise to mimic small-batch gradient noise (ablation) | large-batch training · large-batch experiments | generalization error | No clear effect |
Closest papers by meaning
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.93)
- Understanding deep learning requires rethinking generalization (ICLR 2017 · similarity 0.90)
- Extrapolation for Large-batch Training in Deep Learning (ICML 2020 · similarity 0.90)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.89)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.88)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.