Research map / Optimization / Distributed and compressed
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use blockwise 1-bit compression with error feedback and Nesterov momentum instead of full-precision momentum SGD | Distributed ResNet-50 training · ImageNet, 7 workers | testing accuracy: 76.27% -> 76.77% | Helped |
| Use blockwise 1-bit compression with error feedback and Nesterov momentum instead of full-precision momentum SGD | Distributed ResNet-50 training · ImageNet, 7 workers | time to reach SGDM's highest accuracy: 24 hours -> 13 hours; 46% speedup | Helped |
| Use error-feedback blockwise compression instead of sign-only compression | Distributed training · ImageNet | testing accuracy | Helped |
| Add momentum to dist-EF-blockSGD (ablation) | ResNet-20 · CIFAR-100, mini-batch sizes 16 or 32 per worker, before epoch 100 | testing accuracy convergence | Helped |
| Use signSGD with majority vote instead of error-feedback blockwise compression | ResNet-20 · CIFAR-100 | testing accuracy | Hurt |
| Use signum with majority vote instead of error-feedback blockwise compression | ResNet-50 · ImageNet, 7 and 15 workers | testing accuracy: about 4% worse | Hurt |
| Use 15 workers instead of 7 (ablation) | dist-EF-blockSGDM · ImageNet | speedup over SGDM | Hurt |
Closest papers by meaning
- Error Compensated Distributed SGD Can Be Accelerated (NeurIPS 2021 · similarity 0.92)
- signSGD: Compressed Optimisation for Non-Convex Problems (ICML 2018 · similarity 0.91)
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (NeurIPS 2019 · similarity 0.91)
- On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (ICML 2019 · similarity 0.91)
- A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (NeurIPS 2018 · similarity 0.91)
- Communication-efficient Distributed SGD with Sketching (NeurIPS 2019 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.