Research map / Optimization / Nonconvex and smooth optimization
Coordinate and Adam convergence analysis
Proves convergence under Lojasiewicz and nonuniform smoothness for coordinate descent, Adam, and steepest descent.
188 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Nonconvex and smooth optimization, keyword cluster “coordinate · proximal · bounds”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
39 papers in 2023–24 and 49 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 12.2% to 13.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Stochastic Online AUC Maximization (NeurIPS 2016 · 99 citations)
- 2017 · Variants of RMSProp and Adagrad with Logarithmic Regret Bounds (ICML 2017 · 109 citations)
- 2018 · On the regularization of Wasserstein GANs (ICLR 2018 · 133 citations)
- 2019 · Plug-and-Play Methods Provably Converge with Properly Trained Denoisers (ICML 2019 · 99 citations)
- 2020 · Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity (ICLR 2020 · 83 citations)
- 2021 · Simple Stochastic and Online Gradient Descent Algorithms for Pairwise Learning (NeurIPS 2021 · 10 citations)
- 2022 · On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond (NeurIPS 2022 · 30 citations)
- 2023 · On the Convergence Rates of Policy Gradient Methods (ICML 2023 · 12 citations)
- 2024 · PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping (AAAI 2024 · 22 citations)
- 2025 · Stability and Generalization of Zeroth-Order Decentralized Stochastic Gradient Descent with Changing Topology (AAAI 2025 · 1 citation)
- 2026 · On the Convergence Rate of LoRA Gradient Descent (ICML 2026 · 0 citations)
Earliest papers here
- Stochastic Online AUC Maximization (NeurIPS 2016 · 99 citations)
- Learning Supervised PageRank with Gradient-Based and Gradient-Free Optimization Methods (NeurIPS 2016 · 38 citations)
- Veto-Consensus Multiple Kernel Learning (AAAI 2016 · 30 citations)
What has been tried
138 papers here report 953 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 10%, hurt 16%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace the constant stepsize with Diminish 3 | CIFAR-10 · accuracy | 94.10 ± 0.27 -> 94.75 ± 0.20 | Helped | Fundamental Convergence Analysis of Sharpness-Aware Minimization (NeurIPS 2024 · 3 citations) |
| replace Euclidean regularization with negative entropy regularization | General policy classes satisfying (C⋆, εvgd)-VGD · convergence rate | 1/K^{2/3} -> 1/K^{2/7} | Hurt | Convergence of Policy Mirror Descent Beyond Compatible Function Approximation (ICML 2025 · 0 citations) |
| use the (p,p) variant instead of (p,2) | CIFAR test @200 epoch · Testing ACC (%) | 93.54\std{0.06} -> 93.79\std{0.38} | Helped | Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization (ICML 2025 · 0 citations) |
| Use a last-iterate bound instead of a minimum-suboptimality bound | Convex SGD bound · bound shape match to empirical loss curves | Helped | The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training (ICML 2025 · 0 citations) | |
| Increase the inner-loop steps for Surr-GD | Hidden Matching Pennies · convergence to equilibrium | Helped | Solving hidden monotone variational inequalities with surrogate losses (ICLR 2025 · 0 citations) | |
| rotate the quadratic loss with R₁ | Quadratic loss, 100 steps · final loss | 0.00002 ± 0.00001 -> 0.00066 ± 0.00008 | Hurt | Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · 0 citations) |
| Retrain a single XGBoost model on the weighted mixture instead of ensembling proxy models | ACSIncome and CelebA annotation variants · worst-group accuracy | Helped | MixMax: Distributional Robustness in Function Space via Optimal Data Mixtures (ICLR 2025 · 0 citations) | |
| Choose the midpoint of the coordinate-minimizer interval instead of using the rule that ignores constant affine functions | Constrained convex coordinate descent · cycling | six updates | Hurt | Convergence of Some Convex Message Passing Algorithms to a Fixed Point (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 39 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 4 of 39 papers
- ImageNet · 3 of 39 papers
- OpenWebText · 2 of 39 papers
- MNIST · 2 of 39 papers
Baselines
- SGD · 3 of 39 papers
- gradient descent · 3 of 39 papers
- natural policy gradient · 2 of 39 papers
- SGD with momentum · 2 of 39 papers
- Adam · 2 of 39 papers
- AdamW · 2 of 39 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 88 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 3 papers
- The University of Hong Kong · 3 papers
- Flatiron Institute · 3 papers
- Meta · 3 papers
- University of Pennsylvania · 3 papers
- Carnegie Mellon University · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Tel Aviv University · 3 papers
- Nanjing University · 3 papers
- King Abdullah University of Science and Technology · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Adaptive first-order convex methods (190 papers)
- Stochastic gradient last-iterate analysis (178 papers)
- Acceleration and Nesterov dynamics (91 papers)
- Langevin sampling for nonconvex problems (70 papers)
Elsewhere in the topic
- Nonconvex minimax and constrained methods (100 papers)
- Stochastic convex optimization complexity (94 papers)
- Equilibrium learning in zero-sum games (88 papers)
- Stochastic variance reduction (86 papers)
- Complexity of nonconvex first-order methods (79 papers)
- Penalty and implicit-gradient bilevel methods (65 papers)
- Proximal finite-sum methods (47 papers)
- Stochastic Frank-Wolfe methods (43 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).