Research map / Optimization / Distributed and compressed
Decentralized optimization algorithms
Design and analyze decentralized and distributed optimization methods with faster convergence.
94 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Distributed and compressed, keyword cluster “decentralized · asynchronous · byzantine”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
25 papers in 2023–24 and 17 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 35.2% to 22.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · A Consensus-Based Framework for Distributed Bundle Adjustment (CVPR 2016 · 68 citations)
- 2017 · Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks (ICML 2017 · 192 citations)
- 2018 · Optimal Algorithms for Non-Smooth Distributed Optimization in Networks (NeurIPS 2018 · 76 citations)
- 2019 · On the Complexity of Approximating Wasserstein Barycenters (ICML 2019 · 38 citations)
- 2020 · A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (ICML 2020 · 155 citations)
- 2021 · An Improved Analysis of Gradient Tracking for Decentralized Machine Learning (NeurIPS 2021 · 7 citations)
- 2022 · Decentralized Local Stochastic Extra-Gradient for Variational Inequalities (NeurIPS 2022 · 13 citations)
- 2023 · Epidemic Learning: Boosting Decentralized Learning with Randomized Communication (NeurIPS 2023 · 10 citations)
- 2024 · Jointly Improving the Sample and Communication Complexities in Decentralized Stochastic Minimax Optimization (AAAI 2024 · 5 citations)
- 2025 · Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence (AAAI 2025 · 0 citations)
- 2026 · FAB: A First-Order AB-based Gradient Algorithm for Distributed Bilevel Optimization over Time-Varying Directed Graphs (ICML 2026 · 0 citations)
Earliest papers here
- A Consensus-Based Framework for Distributed Bundle Adjustment (CVPR 2016 · 68 citations)
- Fast ADMM Algorithm for Distributed Optimization with Adaptive Penalty (AAAI 2016 · 57 citations)
- Gossip Dual Averaging for Decentralized Optimization of Pairwise Functions (ICML 2016 · 27 citations)
What has been tried
79 papers here report 507 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 10%, hurt 12%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| increase the BTPP branch size B | Logistic regression with nonconvex regularization · gradient norm | Helped | B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data (NeurIPS 2024 · 1 citation) | |
| Increase the number of random vectors used by the biased ZO estimator | MNIST · accuracy | Helped | Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence (AAAI 2025 · 0 citations) | |
| use randomized instead of vanilla event-based communication | MNIST 80% target accuracy · communication events | 816 -> 629 | Helped | Distributed Event-Based Learning via ADMM (ICML 2025 · 0 citations) |
| Use increasing weights $\alpha_t=t+1$ instead of uniform weights | Heterogeneous SCO · excess loss | Helped | SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization (NeurIPS 2024 · 0 citations) | |
| Add loopless Chebyshev acceleration to SS DSGT | Static network, stochastic noise · iteration complexity | O(\bar{σ}²/(µmε) + √L\bar{σ}/((1 − λ₂(W))µ√ε) + L/((1 − λ₂(W))µε^{1/3})) -> O(\bar{σ}²/(µmε) + √L\bar{σ}/(µ√(1 − λ₂(W))√ | Helped | Double Stochasticity Gazes Faster: Snap-Shot Decentralized Stochastic Gradient Tracking Methods (ICML 2024) |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MNIST · 4 of 11 papers
- CIFAR-10 · 2 of 11 papers
Baselines
- Minibatch SGD · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 42 papers; from OpenAlex and the papers' affiliations).
- Purdue University · 3 papers
- EPFL · 3 papers
- CISPA Helmholtz Center for Information Security · 3 papers
- Xi’an Jiaotong University · 3 papers
- Fudan University · 2 papers
- The Chinese University of Hong Kong · 2 papers
- Rensselaer Polytechnic Institute · 2 papers
- Stanford University · 2 papers
- Moscow Institute of Physics and Technology · 2 papers
- King Abdullah University of Science and Technology · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Local and asynchronous SGD (84 papers)
- Byzantine-robust learning (32 papers)
- Compressed distributed optimization (30 papers)
Elsewhere in the topic
- Gradient quantization and sparsification (57 papers)
- Communication complexity theory (35 papers)
- Error feedback theory (17 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).