Research map / Optimization / Distributed and compressed
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use layer-wise instead of entire-model compression with Random k at 0.1% | Random k · CIFAR-10, AlexNet and ResNet-9 | test accuracy | Helped |
| use layer-wise instead of entire-model compression with Random k at other tested ratios | Random k · CIFAR-10, AlexNet and ResNet-9 | test accuracy | No clear effect |
| use layer-wise instead of entire-model compression | TernGrad · CIFAR-10 and ImageNet; CNN benchmarks | test accuracy | Helped |
| use layer-wise instead of entire-model compression | QSGD · ImageNet | test accuracy: 1.52× better | Helped |
| use layer-wise instead of entire-model compression | Adaptive Threshold · CIFAR-10, AlexNet and ResNet-9 | test accuracy | Helped |
| use Adaptive Threshold compression | Adaptive Threshold · CIFAR-10, AlexNet and ResNet-9 | test accuracy: ≈70% | Hurt |
| use layer-wise instead of entire-model compression across tested thresholds | Threshold v · CIFAR-10, ResNet-9 | test accuracy | No clear effect |
| use layer-wise instead of entire-model compression at ratios above 10% | Top k · CIFAR-10 | test accuracy | No clear effect |
| use entire-model instead of layer-wise compression at ratios ≤ 10% | Top k · CIFAR-10 | test accuracy | Helped |
| use layer-wise instead of entire-model compression | Top k · ImageNet, ResNet-50 | top-5 test accuracy: 1.25-1.63× better | Helped |
| use layer-wise instead of entire-model compression, including at small sparsification ratios | Top k · ImageNet, ResNet-50 | top-5 test accuracy: 1.25-1.63× better | Helped |
| use layer-wise instead of entire-model compression with Top k and Nesterov momentum | Top k with SGD and Nesterov momentum · CIFAR-10, ResNet-9 | test accuracy | No clear effect |
Closest papers by meaning
- Quantized Compressive Sampling of Stochastic Gradients for Efficient Communication in Distributed Deep Learning (AAAI 2020 · similarity 0.92)
- Practical Low-Rank Communication Compression in Decentralized Deep Learning (NeurIPS 2020 · similarity 0.92)
- ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training (NeurIPS 2020 · similarity 0.91)
- Beyond Communication Overhead: A Multilevel Monte Carlo Approach for Mitigating Compression Bias in Distributed Learning (ICML 2025 · similarity 0.90)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.89)
- Accelerated Methods with Compressed Communications for Distributed Optimization Problems Under Data Similarity (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.