AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Measuring the Effects of Data Parallelism on Neural Network Training

Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl

JMLR 2019 · Architectures and efficiency › RNNs and normalization · approach: Efficient network compression and training · 336 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add momentum (Nesterov or heavy-ball) instead of plain SGD (ablation)Simple CNN on MNIST, Transformer on LM1B, ResNet-8 on CIFAR-10 · steps to goal validation error vs batch sizesteps to result: extends perfect scaling region to larger batch sizesImproved
add momentum at small batch sizes (ablation)Simple CNN on MNIST, Transformer on LM1B · small batch sizes within perfect scalingsteps to resultNo clear effect
use Nesterov instead of heavy-ball momentum (ablation)Transformer on LM1B · steps to result vs batch sizesteps to resultNo clear effect
add label smoothing at large batch sizes (ablation)ResNet-50 on ImageNet · ImageNet, fixed compute budget per batch sizevalidation error: improved the error by as much as one percentage point at large batch sizesImproved
add label smoothing at small batch sizes (ablation)ResNet-50 on ImageNet · ImageNetvalidation errorNo clear effect
add label smoothing (ablation)Simple CNN on MNIST and Fashion MNIST · MNIST, Fashion MNISTvalidation errorImproved
increase L2 weight penalty or add Gaussian gradient noise to control overfitting at large batch (ablation)ResNet-50 on ImageNet · ImageNet large batchoverfittingNo clear effect
use ResNet-50 (batch norm, residuals) instead of VGG-11ImageNet · ImageNetmaximum useful batch sizeImproved
use Transformer instead of LSTMLM1B · LM1Bmaximum useful batch sizeImproved
narrow the Transformer (ablation)Transformer on LM1B · LM1Bmaximum useful batch sizeImproved
vary width of Simple CNN (ablation)Simple CNN on MNIST · MNISTsteps to result vs batch sizeNo clear effect
use half the images (random subset) of ImageNet (ablation)ResNet-50 on ImageNet · ImageNet subsetsmaximum useful batch sizeHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy