Research map / Optimization / Nonconvex and smooth optimization
Adaptive first-order convex methods
Gives convergence rates for adaptive and generalized-smoothness first-order methods on convex problems.
190 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Nonconvex and smooth optimization, keyword cluster “coordinate · proximal · bounds”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
36 papers in 2023–24 and 44 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 11.2% to 12.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · On Graduated Optimization for Stochastic Non-Convex Problems (ICML 2016 · 54 citations)
- 2017 · Prox-PDA: The Proximal Primal-Dual Algorithm for Fast Distributed Nonconvex Optimization and Learning Over Networks (ICML 2017 · 114 citations)
- 2018 · Convergence guarantees for a class of non-convex and non-smooth optimization problems (ICML 2018 · 43 citations)
- 2019 · A Linearly Convergent Proximal Gradient Algorithm for Decentralized Optimization (NeurIPS 2019 · 49 citations)
- 2020 · Finite-Time Convergence in Continuous-Time Optimization (ICML 2020 · 21 citations)
- 2021 · Rotation Coordinate Descent for Fast Globally Optimal Rotation Averaging (CVPR 2021 · 12 citations)
- 2022 · Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling (NeurIPS 2022 · 8 citations)
- 2023 · Accelerated Primal-Dual Methods for Convex-Strongly-Concave Saddle Point Problems (ICML 2023 · 5 citations)
- 2024 · Adaptive Proximal Gradient Method for Convex Optimization (NeurIPS 2024 · 13 citations)
- 2025 · An Enhanced Levenberg--Marquardt Method via Gram Reduction (AAAI 2025 · 1 citation)
- 2026 · Randomized Feasibility Methods for Constrained Optimization with Adaptive Step Sizes (ICML 2026 · 0 citations)
Earliest papers here
- On Graduated Optimization for Stochastic Non-Convex Problems (ICML 2016 · 54 citations)
- The non-convex Burer-Monteiro approach works on smooth semidefinite programs (NeurIPS 2016 · 33 citations)
- SDCA without Duality, Regularization, and Individual Convexity (ICML 2016 · 32 citations)
What has been tried
154 papers here report 1,014 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 11%, hurt 14%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Initialize B_0 = LI rather than B_0 = μI | Strongly convex, L-smooth functions · iterations to reach the stated linear phase | 1 -> d\log\kappa | Helped | Non-asymptotic Global Convergence Analysis of BFGS with the Armijo-Wolfe Line Search (NeurIPS 2024 · 2 citations) |
| Vary the Gram-matrix reuse interval m among 1, 50, 100, and 500 | Chandrasekhar H-equation (N = 100, N = 200, and N = 300) · iteration count | Mixed | An Enhanced Levenberg--Marquardt Method via Gram Reduction (AAAI 2025 · 1 citation) | |
| replace global min-consensus with local min-consensus | Ridge regression on line and Erdős–Rényi graphs · convergence speed | No clear effect | Achieving Linear Convergence with Parameter-Free Algorithms in Decentralized Optimization (NeurIPS 2024 · 1 citation) | |
| Use q=1.0 and p=1.6 instead of the original q=1, p=2 setting | CSI300 OPS · cumulative wealth | 1.7750 -> 2.0932 | Helped | De-singularity Subgradient for the q-th-Powered lₚ-Norm Weber Location Problem (AAAI 2025 · 0 citations) |
| Decrease \rho toward 0 | Deterministic sparse optimization with support-preserving constraints · objective upper bound | (1+2 \rho) R(\bar{\vw})+ \varepsilon | Helped | Optimization over Sparse Support-Preserving Sets: Two-Step Projection with Global Optimality Guarantees (ICML 2025 · 0 citations) |
| Use multiple gossip steps instead of a single gossip step | MNIST · training loss | Helped | Achieving Linear Speedup and Near-Optimal Complexity for Decentralized Optimization over Row-stochastic Networks (ICML 2025 · 0 citations) | |
| Replace ℓ(s) = 800 + 2s² with ℓ(s) = 800 + 2s | f(x) = -log x - log(0.1 - x) · convergence | Hurt | Toward a Unified Theory of Gradient Descent under Generalized Smoothness (ICML 2025 · 0 citations) | |
| Initialize BFGS with cI instead of I | Hard cubic function · convergence | No clear effect | Affine-Invariant Global Non-Asymptotic Convergence Analysis of BFGS under Self-Concordance (NeurIPS 2025 · 0 citations) | |
| Use (c_0, c_1) = (10, 10) among the adaptive step-size choices | Monotone problem · relative error | Helped | Extragradient Method for $(L_0, L_1)$-Lipschitz Root-finding Problems (NeurIPS 2025 · 0 citations) | |
| Add an integral controller term | Quadratic programming with approximate inverse · convergence to optimal solution | Helped | Constrained Optimization From a Control Perspective via Feedback Linearization (NeurIPS 2025 · 0 citations) | |
| Use CONGO-E instead of CONGO-Z, replacing the Bernoulli measurement matrix with a Gaussian matrix | Jackson network simulation · cost | Helped | CONGO: Compressive Online Gradient Optimization (ICLR 2025 · 0 citations) | |
| Remove the dual-variable updates and smoothing by fixing z=0 and mu=0 | Sparse FDA · objective value after 20 seconds | Hurt | ADMM for Structured Fractional Minimization (ICLR 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 29 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 6 of 29 papers
- MNIST · 5 of 29 papers
- Boston housing · 2 of 29 papers
- MovieLens 100K · 2 of 29 papers
Baselines
- gradient descent · 8 of 29 papers
- Adam · 3 of 29 papers
- SGD · 2 of 29 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 80 papers; from OpenAlex and the papers' affiliations).
- KU Leuven · 4 papers
- University of Texas, Austin · 4 papers
- University of Texas at Austin · 4 papers
- Fudan University · 3 papers
- University of California, Berkeley · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Moscow Institute of Physics and Technology · 3 papers
- The Chinese University of Hong Kong · 2 papers
- Arizona State University · 2 papers
- Inria · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Stochastic gradient last-iterate analysis (178 papers)
- Coordinate and Adam convergence analysis (188 papers)
- Acceleration and Nesterov dynamics (91 papers)
- Langevin sampling for nonconvex problems (70 papers)
Elsewhere in the topic
- Nonconvex minimax and constrained methods (100 papers)
- Stochastic convex optimization complexity (94 papers)
- Equilibrium learning in zero-sum games (88 papers)
- Stochastic variance reduction (86 papers)
- Complexity of nonconvex first-order methods (79 papers)
- Penalty and implicit-gradient bilevel methods (65 papers)
- Proximal finite-sum methods (47 papers)
- Stochastic Frank-Wolfe methods (43 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).