AI Paper Atlas

Research map / Federated learning and privacy / Communication-efficient FL

Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices

Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, Gennady Pekhimenko

NeurIPS 2021 · Federated learning and privacy › Communication-efficient FL · approach: Decentralized peer-to-peer training · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace gossip-based PushSum averaging with Moshpit All-ReduceSGP-style distributed SGD · ImageNet, ResNet-50, heterogeneoustraining speed: 1.3 times fasterImproved
Use Moshpit All-Reduce for distributed averagingALBERT-large pretraining · BookCorpus, preemptible cloud instancestraining speed: 1.5 times fasterImproved
Use Moshpit averaging instead of PushSumSGP-style distributed SGD · ImageNet, ResNet-50, homogeneoustraining efficiencyImproved
Use Moshpit averaging instead of PushSumSGP-style distributed SGD · ImageNet, ResNet-50, heterogeneoustraining efficiency: over 30%Improved
Use conventional All-Reduce instead of Moshpit SGD under network latencyImageNet distributed training · ImageNet, ResNet-50, homogeneoustraining time: 1.5× of Moshpit SGDHurt
Use simple parameter averaging without pseudo-gradient recovery (ablation)Moshpit SGD with LAMB · BookCorpus, ALBERT-largeconvergenceHurt
Use random-group averaging instead of Moshpit's structured matchmaking (ablation)decentralized averaging · 512–1024 peers, low grid utilizationaverage squared distance to global averageNo clear effect
Use random-group averaging instead of Moshpit's structured matchmaking (ablation)decentralized averaging · 512 peers, higher grid utilizationaverage squared distance to global averageHurt
Use standard All-Reduce with restarts when peer failures occurdecentralized averaging · peer failure probability at least 0.005iterations to average: 0.005Hurt
Use gossip or PushSum instead of Moshpit All-Reducedecentralized averaging · 512–1024 peersiterations to convergeHurt
Train with AD-PSGD instead of Moshpit SGDResNet-50 distributed training · ImageNettop-1 validation accuracy: 150 epochsHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy