Research map / Optimization / Distributed and compressed
Communication-efficient Distributed SGD with Sketching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SKETCHED-SGD with 40x compression instead of uncompressed distributed SGD | Transformer · WMT 2014 English to German test | BLEU: 26.29 -> 26.796 | Helped |
| Use SKETCHED-SGD with 20x compression instead of uncompressed distributed SGD | Transformer · WMT 2014 English to German test | BLEU: 26.29 -> 26.875 | Helped |
| Use SKETCHED-SGD with 40x compression instead of uncompressed distributed SGD | LSTM · WMT 2014 English to German test | BLEU: 20.87 -> 20.95 | No clear effect |
| Use top-k SGD with k = 100,000 instead of uncompressed distributed SGD | Transformer · WMT 2014 English to German test | BLEU: 26.29 -> 26.65 | Helped |
| Use top-k SGD with k = 100,000 instead of uncompressed distributed SGD | LSTM · WMT 2014 English to German test | BLEU: 20.87 -> 22.2 | Helped |
| Increase compression from 17x to 11x by training 50% more iterations | SKETCHED-SGD residual network · CIFAR-10 test | accuracy: 92.5% -> 94% | Helped |
| Increase compression rate (ablation) | SKETCHED-SGD residual network · CIFAR-10 test | accuracy | Hurt |
| Use a very large sketch size relative to Pk (ablation) | SKETCHED-SGD residual network · CIFAR-10 test | accuracy | Hurt |
| Reduce sketch size |S| or P to extremely small values (ablation) | SKETCHED-SGD transformer · WMT 2014 English to German | BLEU | Hurt |
| Use a larger k (ablation) | Top-k SGD · WMT 2014 English to German and CIFAR-10 | test performance | Mixed |
| Scale worker count up to W = 256 (ablation) | SKETCHED-SGD and local top-k · CIFAR-10 | test accuracy | No clear effect |
Closest papers by meaning
- signSGD: Compressed Optimisation for Non-Convex Problems (ICML 2018 · similarity 0.91)
- Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback (NeurIPS 2019 · similarity 0.91)
- Local SGD Converges Fast and Communicates Little (ICLR 2019 · similarity 0.90)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.90)
- Sketching for Distributed Deep Learning: A Sharper Analysis (NeurIPS 2024 · similarity 0.89)
- Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.