Research map / Optimization / Distributed and compressed
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Quantize AlexNet gradients to 4 bits | AlexNet · ImageNet | top-1 accuracy: 59.50% -> 60.05% | Helped |
| Quantize AlexNet gradients to 4 bits | AlexNet · ImageNet | speedup: 2.05 × | Helped |
| Quantize ResNet152 gradients to 8 bits | ResNet152 · ImageNet | top-1 accuracy: 77.0% -> 76.74% | No clear effect |
| Quantize ResNet152 gradients to 8 bits | ResNet152 · ImageNet | speedup: 1.56 × | Helped |
| Quantize ResNet50 gradients to 4 bits | ResNet50 · ImageNet | top-1 accuracy: 74.68% -> 74.76% | Helped |
| Quantize ResNet110 gradients to 4 bits | ResNet110 · CIFAR-10 | top-1 accuracy: 93.86% -> 94.19% | Helped |
| Quantize LSTM gradients to 4 bits | LSTM · AN4 | top-1 accuracy: 81.13% -> 81.15% | Helped |
| Use 4-bit QSGD with bucket size 8192 instead of less aggressive quantization (ablation) | AlexNet · ImageNet | top-1 accuracy: 0.68% | Hurt |
| Use 4-bit QSGD with bucket size 8192 instead of less aggressive quantization (ablation) | AlexNet · ImageNet | top-5 accuracy: 0.57% | Hurt |
| Use 2-bit QSGD with bucket size 64 (ablation) | AlexNet · ImageNet | top-1 accuracy: 1.73% | Hurt |
| Use 2-bit QSGD with bucket size 64 (ablation) | AlexNet · ImageNet | top-1 accuracy: 1.18% | Hurt |
| Quantize convolutional layers aggressively at 2-bit precision (ablation) | Vision networks · ImageNet | accuracy | Hurt |
Closest papers by meaning
- Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations (NeurIPS 2019 · similarity 0.92)
- A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (NeurIPS 2018 · similarity 0.92)
- Adaptive Gradient Quantization for Data-Parallel SGD (NeurIPS 2020 · similarity 0.91)
- Communication Efficient SGD via Gradient Sampling With Bayes Prior (CVPR 2021 · similarity 0.91)
- Quantized Compressive Sampling of Stochastic Gradients for Efficient Communication in Distributed Deep Learning (AAAI 2020 · similarity 0.90)
- DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated Compression (ICML 2019 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.