Distributed and compressed: research map
388 papers on Distributed and compressed in Optimization, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 2 clusters and 7 approaches. The busiest year so far is 2021.
What it is
Training on many machines is limited by communication. This topic designs and analyses methods that send less (compressed or quantised gradients), sync less often (local and asynchronous SGD), have no central server (decentralised) or tolerate faulty and malicious workers (Byzantine robustness).
Terms you will see on this page:
- Gradient compression
- sending fewer or lower-precision gradient values.
- Local SGD
- workers take several steps alone before averaging.
- Byzantine robustness
- still converging when some workers send arbitrary wrong updates.
Start reading with the most cited papers here:
- On the Convergence of FedAvg on Non-IID Data (ICLR 2020 · 979 citations)
- Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent (NeurIPS 2017 · 967 citations)
Within Optimization, its share held steady from 8.2% in 2023–24 to 7.7% in 2025–26 (71 → 75 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Distributed and compressed in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Local and asynchronous SGD rising: 18% → 28% of the topic (13 → 21 papers)
- Gradient quantization and sparsification rising: 8% → 12% of the topic (6 → 9 papers)
- Byzantine-robust learning rising: 8% → 11% of the topic (6 → 8 papers)
- Decentralized optimization algorithms falling: 35% → 23% of the topic (25 → 17 papers)
- Compressed distributed optimization falling: 14% → 9% of the topic (10 → 7 papers)
Papers from 2025–2026 that are already cited most:
- AdaGK-SGD: Adaptive Global Knowledge Guided Distributed Stochastic Gradient Descent (AAAI 2025 · 1 citations)
- Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · 1 citations)
- On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach (ICML 2026 · 0 citations)
- A Tight Theory of Error Feedback Algorithms in Distributed Optimization (ICML 2026 · 0 citations)
Approaches and key papers
Decentralized and robust distributed training · 273 papers
Keywords: decentralized, asynchronous, byzantine
Approaches in this cluster:
- Decentralized optimization algorithms (113 papers)
Design and analyze decentralized and distributed optimization methods with faster convergence. - Local and asynchronous SGD (90 papers)
Analyze asynchronous, local, and minibatch SGD for communication-efficient distributed training. - Byzantine-robust learning (35 papers)
Defend distributed training against adversarial workers with robust aggregation. - Compressed distributed optimization (35 papers)
Reduce communication in decentralized and distributed training via compression and error feedback.
Most cited and most cited since 2024:
- On the Convergence of FedAvg on Non-IID Data (ICLR 2020 · 979 citations)
- Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent (NeurIPS 2017 · 967 citations)
- Byzantine-Robust Decentralized Learning via Remove-then-Clip Aggregation (AAAI 2024 · 9 citations)
- Jointly Improving the Sample and Communication Complexities in Decentralized Stochastic Minimax Optimization (AAAI 2024 · 5 citations)
Communication-efficient training · 115 papers
Keywords: communication distributed, gradient compression, error feedback
Approaches in this cluster:
- Gradient quantization and sparsification (59 papers)
Compress gradients with quantization and sparsification to cut communication in distributed SGD. - Communication complexity theory (37 papers)
Prove lower bounds and communication-optimal distributed algorithms with compression. - Error feedback theory (19 papers)
Analyze error feedback mechanisms for compressed distributed optimization.
Most cited and most cited since 2024:
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · 911 citations)
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (NeurIPS 2017 · 662 citations)
- Exponential Quantum Communication Advantage in Distributed Inference and Learning (NeurIPS 2024 · 2 citations)
- Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · 1 citations)
Related topics in Optimization
- Implicit bias and generalization (1,070)
- Nonconvex and smooth optimization (1,707)
- Optimizers (1,206)
