Research map / Optimization / Distributed and compressed
Gradient quantization and sparsification
Compress gradients with quantization and sparsification to cut communication in distributed SGD.
57 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Distributed and compressed, keyword cluster “communication distributed · gradient compression · error feedback”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
6 papers in 2023–24 and 9 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 8.5% to 12.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · 911 citations)
- 2018 · Sparsified SGD with Memory (NeurIPS 2018 · 221 citations)
- 2019 · Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations (NeurIPS 2019 · 228 citations)
- 2020 · SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (ICLR 2020 · 64 citations)
- 2021 · Rethinking gradient sparsification as total error minimization (NeurIPS 2021 · 18 citations)
- 2022 · A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks (NeurIPS 2022 · 6 citations)
- 2023 · Quantized Distributed Training of Large Models with Convergence Guarantees (ICML 2023 · 5 citations)
- 2024 · Exponential Quantum Communication Advantage in Distributed Inference and Learning (NeurIPS 2024 · 2 citations)
- 2025 · Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · 1 citation)
- 2026 · DeMo: Decoupled Momentum Optimization (ICLR 2026 · 0 citations)
Earliest papers here
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · 911 citations)
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (NeurIPS 2017 · 662 citations)
- Sparsified SGD with Memory (NeurIPS 2018 · 221 citations)
What has been tried
50 papers here report 467 measured changes (a component added, removed or swapped, compared with a baseline): helped 57%, no clear effect 20%, hurt 19%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace PReLU with a degree-2 polynomial | ogbn-products node classification · accuracy | 79.48 ± 0.07 -> 78.51 ± 0.05 | Hurt | Exponential Quantum Communication Advantage in Distributed Inference and Learning (NeurIPS 2024 · 2 citations) |
| increase the sketch size | CIFAR-10 training · training error | Helped | Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · 1 citation) | |
| Replace ACCO with delayed parameter updates (DPU), without warmup | TinyStories · training loss | Hurt | ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · 0 citations) | |
| Increase the QSR minimum synchronization period from H_min=2 to H_min=4 | ImageNet, batch size 16384, cosine · validation accuracy (%) | 79.27 -> 78.65 | Hurt | A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · 0 citations) |
| Add DMGC to DPGS | C16-Multiscale · time (s) | 544 ±37 to 142 ±11 | Helped | Distributed Parallel Gradient Stacking(DPGS): Solving Whole Slide Image Stacking Challenge in Multi-Instance Learning (ICML 2025) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 15 papers; from OpenAlex and the papers' affiliations).
- National University of Defense Technology · 2 papers
Use it
Download BibTeX of 31 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Communication complexity theory (35 papers)
- Error feedback theory (17 papers)
Elsewhere in the topic
- Decentralized optimization algorithms (94 papers)
- Local and asynchronous SGD (84 papers)
- Byzantine-robust learning (32 papers)
- Compressed distributed optimization (30 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).