Research map / Optimization / Distributed and compressed
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SlowMo to Local SGD | Local SGD · CIFAR-10 | validation accuracy: 91.73% -> 93.20% | Helped |
| Add SlowMo to OSGP | OSGP · CIFAR-10 | validation accuracy: 93.17% -> 93.74% | Helped |
| Add SlowMo to SGP | SGP · CIFAR-10 | validation accuracy: 93.90% -> 94.32% | Helped |
| Add SlowMo to Local SGD | Local SGD · ImageNet | validation accuracy: 69.94% -> 73.24% | Helped |
| Add SlowMo to OSGP | OSGP · ImageNet | validation accuracy: 74.96% -> 75.54% | Helped |
| Add SlowMo to SGP | SGP · ImageNet | validation accuracy: 75.15% -> 75.73% | Helped |
| Add SlowMo to Local Adam | Local Adam · WMT'16 En-De | BLEU: 26.62 -> 27.14 | Helped |
| Add SlowMo to SGP | SGP · WMT'16 En-De | BLEU: 26.92 -> 27.84 | Helped |
| Skip exact AllReduce averaging during the SlowMo update (ablation) | SGP + SlowMo · ImageNet | validation accuracy: 75.73% -> 75.78% | No clear effect |
| Skip exact AllReduce averaging during the SlowMo update (ablation) | SGP + SlowMo · WMT'16 En-De | validation NLL: 2.10 -> 2.11 | Hurt |
| Average Adam buffers across workers instead of maintaining them (ablation) | SlowMo with Local Adam · WMT'16 En-De | validation loss: 2.106 -> 2.101 | Helped |
| Reset Adam buffers to zero instead of maintaining them (ablation) | SlowMo with Local Adam · WMT'16 En-De | validation loss: 2.106 -> 4.732 | Hurt |
Closest papers by meaning
- On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (ICML 2019 · similarity 0.90)
- SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization (NeurIPS 2024 · similarity 0.90)
- Communication trade-offs for Local-SGD with large step size (NeurIPS 2019 · similarity 0.89)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.89)
- Ordered Momentum for Asynchronous SGD (NeurIPS 2024 · similarity 0.88)
- Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.