Research map / Architectures and efficiency / RNNs and normalization
Measuring the Effects of Data Parallelism on Neural Network Training
Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add momentum (Nesterov or heavy-ball) instead of plain SGD (ablation) | Simple CNN on MNIST, Transformer on LM1B, ResNet-8 on CIFAR-10 · steps to goal validation error vs batch size | steps to result: extends perfect scaling region to larger batch sizes | Improved |
| add momentum at small batch sizes (ablation) | Simple CNN on MNIST, Transformer on LM1B · small batch sizes within perfect scaling | steps to result | No clear effect |
| use Nesterov instead of heavy-ball momentum (ablation) | Transformer on LM1B · steps to result vs batch size | steps to result | No clear effect |
| add label smoothing at large batch sizes (ablation) | ResNet-50 on ImageNet · ImageNet, fixed compute budget per batch size | validation error: improved the error by as much as one percentage point at large batch sizes | Improved |
| add label smoothing at small batch sizes (ablation) | ResNet-50 on ImageNet · ImageNet | validation error | No clear effect |
| add label smoothing (ablation) | Simple CNN on MNIST and Fashion MNIST · MNIST, Fashion MNIST | validation error | Improved |
| increase L2 weight penalty or add Gaussian gradient noise to control overfitting at large batch (ablation) | ResNet-50 on ImageNet · ImageNet large batch | overfitting | No clear effect |
| use ResNet-50 (batch norm, residuals) instead of VGG-11 | ImageNet · ImageNet | maximum useful batch size | Improved |
| use Transformer instead of LSTM | LM1B · LM1B | maximum useful batch size | Improved |
| narrow the Transformer (ablation) | Transformer on LM1B · LM1B | maximum useful batch size | Improved |
| vary width of Simple CNN (ablation) | Simple CNN on MNIST · MNIST | steps to result vs batch size | No clear effect |
| use half the images (random subset) of ImageNet (ablation) | ResNet-50 on ImageNet · ImageNet subsets | maximum useful batch size | Hurt |
Closest papers by meaning
- How Does Critical Batch Size Scale in Pre-training? (ICLR 2025 · similarity 0.90)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.87)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.87)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.87)
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training (NeurIPS 2025 · similarity 0.86)
- Accelerating neural network training: An analysis of the AlgoPerf competition (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
