Research map / Federated learning and privacy / Communication-efficient FL
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, Gennady Pekhimenko
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace gossip-based PushSum averaging with Moshpit All-Reduce | SGP-style distributed SGD · ImageNet, ResNet-50, heterogeneous | training speed: 1.3 times faster | Improved |
| Use Moshpit All-Reduce for distributed averaging | ALBERT-large pretraining · BookCorpus, preemptible cloud instances | training speed: 1.5 times faster | Improved |
| Use Moshpit averaging instead of PushSum | SGP-style distributed SGD · ImageNet, ResNet-50, homogeneous | training efficiency | Improved |
| Use Moshpit averaging instead of PushSum | SGP-style distributed SGD · ImageNet, ResNet-50, heterogeneous | training efficiency: over 30% | Improved |
| Use conventional All-Reduce instead of Moshpit SGD under network latency | ImageNet distributed training · ImageNet, ResNet-50, homogeneous | training time: 1.5× of Moshpit SGD | Hurt |
| Use simple parameter averaging without pseudo-gradient recovery (ablation) | Moshpit SGD with LAMB · BookCorpus, ALBERT-large | convergence | Hurt |
| Use random-group averaging instead of Moshpit's structured matchmaking (ablation) | decentralized averaging · 512–1024 peers, low grid utilization | average squared distance to global average | No clear effect |
| Use random-group averaging instead of Moshpit's structured matchmaking (ablation) | decentralized averaging · 512 peers, higher grid utilization | average squared distance to global average | Hurt |
| Use standard All-Reduce with restarts when peer failures occur | decentralized averaging · peer failure probability at least 0.005 | iterations to average: 0.005 | Hurt |
| Use gossip or PushSum instead of Moshpit All-Reduce | decentralized averaging · 512–1024 peers | iterations to converge | Hurt |
| Train with AD-PSGD instead of Moshpit SGD | ResNet-50 distributed training · ImageNet | top-1 validation accuracy: 150 epochs | Hurt |
Closest papers by meaning
- DropCompute: simple and more robust distributed synchronous training via compute variance reduction (NeurIPS 2023 · similarity 0.88)
- $\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning (NeurIPS 2023 · similarity 0.88)
- Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (NeurIPS 2018 · similarity 0.88)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.87)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.86)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
