Research map / Optimization / Distributed and compressed
Asynchronous Decentralized SGD with Quantized and Local Updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SwarmSGD instead of LB-SGD | Transformer-XL · WMT17 (En-Ge), 16 nodes, 10 epochs | BLEU: 26.1 -> 26.17 | Helped |
| Use SwarmSGD instead of LB-SGD | Transformer-XL · WMT17 (En-Ge), 16 nodes | end-to-end speedup: $\sim 1.5\times$ | Helped |
| Run SwarmSGD for 5 additional epochs (ablation) | SwarmSGD · WMT17 (En-Ge), 32 nodes | BLEU: $\sim 23.5$ -> $\sim 25.72$ | Helped |
| Increase SwarmSGD local steps to 4 (ablation) | SwarmSGD on ResNet18 · ImageNet | Top-1 accuracy recovery | No clear effect |
| Apply 8-bit model quantization (ablation) | SwarmSGD · Image classification tasks | accuracy recovery | No clear effect |
| Evaluate the real average of all models rather than an arbitrary individual model (ablation) | SwarmSGD · Image classification training | accuracy | No clear effect |
| Add SlowMo (ablation) | decentralized training methods including SwarmSGD · CIFAR-10 | accuracy | Helped |
| Add SlowMo (ablation) | decentralized training methods including SwarmSGD · ImageNet | accuracy | No clear effect |
| Use SwarmSGD with the reported local-step schedules instead of the baseline epoch schedules | ResNet20, ResNet18, and ResNet50 · CIFAR-10 and ImageNet | epochs needed for approximate Top-1 accuracy recovery: 200 epochs -> 300 epochs; 90 epochs -> 240 epochs | Mixed |
Closest papers by meaning
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication (ICML 2019 · similarity 0.91)
- A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (ICML 2020 · similarity 0.91)
- Communication trade-offs for Local-SGD with large step size (NeurIPS 2019 · similarity 0.90)
- Stability-Based Generalization Analysis of the Asynchronous Decentralized SGD (AAAI 2023 · similarity 0.90)
- Scalable Decentralized Learning with Teleportation (ICLR 2025 · similarity 0.89)
- Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.