Research map / Optimization / Distributed and compressed
Don't Use Large Mini-batches, Use Local SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace large-batch training with post-local SGD (H=32) | ResNet-20 · CIFAR-10 | top-1 test accuracy: 92.48 -> 93.02 | Helped |
| replace large-batch training with post-local SGD (H=32) | ResNet-20 · CIFAR-100 | top-1 test accuracy: 68.17 -> 69.38 | Helped |
| replace large-batch training with post-local SGD (H=32) | DenseNet-40-12 · CIFAR-10 | top-1 test accuracy: 94.36 -> 94.58 | Helped |
| replace large-batch training with post-local SGD (H=32) | DenseNet-40-12 · CIFAR-100 | top-1 test accuracy: 74.08 -> 75.03 | Helped |
| replace large-batch training with post-local SGD (H=16) | WideResNet-28-10 · CIFAR-10 | top-1 test accuracy: 95.43 -> 95.94 | Helped |
| replace large-batch training with post-local SGD (H=32) | WideResNet-28-10 · CIFAR-100 | top-1 test accuracy: 79.31 -> 80.65 | Helped |
| add post-local SGD to LARS | ResNet-50 · ImageNet, K Bl=8,192 | top-1 test accuracy: 75.99 -> 76.13 | Helped |
| add post-local SGD to LARS | ResNet-50 · ImageNet, K Bl=16,384 | top-1 test accuracy: 74.52 -> 75.23 | Helped |
| increase local steps from H=1 to H=32 (ablation) | signSGD · CIFAR-10 | top-1 test accuracy: 91.61 -> 92.22 | Helped |
| increase local steps from H=1 to H=64 (ablation) | signSGD · CIFAR-100 | top-1 test accuracy: 67.21 -> 68.12 | Helped |
| add global momentum 0.1 to local momentum 0.9 (ablation) | Local SGD · CIFAR-10 | test top-1: 92.41 -> 92.22 | Hurt |
| add global momentum 0.2 to local momentum 0.9 (ablation) | Local SGD · CIFAR-10 | test top-1: 92.41 -> 92.09 | Hurt |
Closest papers by meaning
- Local SGD Converges Fast and Communicates Little (ICLR 2019 · similarity 0.95)
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.92)
- The Effect of Network Width on the Performance of Large-batch Training (NeurIPS 2018 · similarity 0.91)
- Stochastic Training is Not Necessary for Generalization (ICLR 2022 · similarity 0.91)
- Why (and When) does Local SGD Generalize Better than SGD? (ICLR 2023 · similarity 0.90)
- SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs (NeurIPS 2021 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.