Research map / Optimization / Distributed and compressed
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| periodic quantized averaging (PQASGD) instead of full-communication SGD | distributed SGD · CIFAR-10 ResNet110 16 machines | wall-clock speedup: 2.7x | Helped |
| PQASGD instead of full-communication SGD | distributed SGD · CIFAR-10 ResNet110 | test error: 0.0635 | No clear effect |
| periodic or rotate sparse averaging (p=2..8) instead of full averaging | distributed SGD · CIFAR-10 ResNet32 8 machines | test accuracy | No clear effect |
| partitioned instead of shared training data with QSGD s=sqrt(N)/2 | QSGD · CIFAR-10 VGG16 | training loss convergence | Hurt |
| raise QSGD level to sqrt(mN)/2 on partitioned data | QSGD-P · CIFAR-10 VGG16 | training loss convergence | Helped |
| TernGrad ternary quantization instead of PQASGD | distributed SGD · CIFAR-10 VGG16 | training loss: 0.02 | Hurt |
| combine sparsification with quantization (PQASGD) instead of QSGD alone | QSGD · CIFAR-10 VGG16 shared data | compression efficiency: 0.231 -> 0.030 | Helped |
Closest papers by meaning
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · similarity 0.92)
- Quantized Compressive Sampling of Stochastic Gradients for Efficient Communication in Distributed Deep Learning (AAAI 2020 · similarity 0.92)
- Double Quantization for Communication-Efficient Distributed Optimization (NeurIPS 2019 · similarity 0.91)
- Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback (NeurIPS 2019 · similarity 0.91)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.89)
- JointSQ: Joint Sparsification-Quantization for Distributed Learning (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.