Research map / Optimization / Implicit bias and generalization
Why (and When) does Local SGD Generalize Better than SGD?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add local steps with H=750 | SGD-pretrained model · CIFAR-10 | test accuracy: +1.1% | Helped |
| Add local steps with H=26 | SGD-pretrained model · ImageNet | test accuracy: +0.3% | Helped |
| Use a small learning rate (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | Helped |
| Use a large learning rate (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | Hurt |
| Train for sufficient time (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | Helped |
| Use insufficient training time (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | Hurt |
| Remove SGD pretraining and start from random initialization (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | No clear effect |
| Remove Nesterov momentum and learning-rate decay (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | No clear effect |
| Tune H according to the learning rate (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | Helped |
| Increase H beyond its useful range (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | Mixed |
| Increase H from 24 to 900 after learning-rate decay (ablation) | Local SGD · CIFAR-10 | test accuracy: 24 to 900 | Helped |
| Increase worker count K while keeping η and H fixed (ablation) | Local SGD · CIFAR-10 and ImageNet | test accuracy | Helped |
Closest papers by meaning
- Don't Use Large Mini-batches, Use Local SGD (ICLR 2020 · similarity 0.90)
- Local SGD Converges Fast and Communicates Little (ICLR 2019 · similarity 0.90)
- SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs (NeurIPS 2021 · similarity 0.89)
- Communication-efficient SGD: From Local SGD to One-Shot Averaging (NeurIPS 2021 · similarity 0.88)
- Communication trade-offs for Local-SGD with large step size (NeurIPS 2019 · similarity 0.88)
- An Alternative View: When Does SGD Escape Local Minima? (ICML 2018 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.