Research map / Optimization / Distributed and compressed
DecentLaM: Decentralized Momentum SGD for Large-Batch Deep Training
Kun Yuan, Yiming Chen, Xinmeng Huang, Yingya Zhang, Pan Pan, Yinghui Xu, Wotao Yin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DecentLaM instead of DmSGD at batch 32k | decentralized momentum SGD · ImageNet-1K val (batch 32k) | top-1 accuracy (%): 74.97 -> 76.22 | Improved |
| use DecentLaM instead of DmSGD at batch 2k | decentralized momentum SGD · ImageNet-1K val (batch 2k) | top-1 accuracy (%): 76.27 -> 76.43 | No clear effect |
| use DecentLaM instead of parallel PmSGD at batch 32k | PmSGD · ImageNet-1K val (batch 32k) | top-1 accuracy (%): 75.27 -> 76.22 | Improved |
| increase batch size from 2K to 32K in DmSGD without LARS | DmSGD · ImageNet-1K val (Table 1 setup) | top-1 accuracy (%): 76.5% -> 74.9% | Hurt |
| increase batch size from 2K to 8K in DmSGD on CIFAR-10 | DmSGD · CIFAR-10 (ResNet-20) | top-1 accuracy (%): 91.5% -> 88.3% | Hurt |
| increase batch size from 16k to 32k in DecentLaM | DecentLaM · ImageNet-1K val (ResNet-50) | top-1 accuracy (%): 76.73 -> 76.22 | Hurt |
| use ring instead of symmetric exponential topology at batch 32k (ablation) | DecentLaM · ImageNet-1K val (ResNet-50, batch 32k) | top-1 accuracy (%): 76.22 -> 76.34 | Improved |
| use mesh instead of symmetric exponential topology at batch 32k (ablation) | DecentLaM · ImageNet-1K val (ResNet-50, batch 32k) | top-1 accuracy (%): 76.22 -> 76.47 | No clear effect |
| use bipartite random match instead of symmetric exponential topology at batch 16k (ablation) | DecentLaM · ImageNet-1K val (ResNet-50, batch 16k) | top-1 accuracy (%): 76.73 -> 76.53 | No clear effect |
| use DecentLaM instead of DmSGD on COCO with Faster R-CNN | DmSGD · COCO | mAP: 36.4 -> 37.1 | Improved |
| use DecentLaM instead of DmSGD on PASCAL VOC with RetinaNet | DmSGD · PASCAL VOC | mAP: 79.1 -> 79.3 | No clear effect |
| use DecentLaM instead of D2-DmSGD at batch 2k | decentralized momentum SGD · ImageNet-1K val (batch 2k) | top-1 accuracy (%): 75.44 -> 76.43 | Improved |
Closest papers by meaning
- Quasi-global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data (ICML 2021 · similarity 0.88)
- DPA: Decentralized Primal Averaging with Quasi-Global Momentum for Highly Heterogeneous Data (NeurIPS 2026 · similarity 0.88)
- Ordered Momentum for Asynchronous SGD (NeurIPS 2024 · similarity 0.88)
- Decentralized SGD and Average-direction SAM are Asymptotically Equivalent (ICML 2023 · similarity 0.87)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.87)
- A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
