Optimizers: research map
1,206 papers on Optimizers in Optimization, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 4 clusters and 16 approaches. The busiest year so far is 2025.
What it is
The algorithms that actually train networks, such as Adam and SGD with momentum, and how to make them faster, more memory-efficient or better understood. The topic also covers gradients through inner optimisation (bilevel and meta-learning), sampling-based optimisers and zeroth-order methods.
Terms you will see on this page:
- Adam
- an adaptive optimizer that scales each parameter's step by running gradient statistics.
- Preconditioning
- reshaping gradient steps using curvature information.
- Bilevel optimization
- an optimisation problem nested inside another, as in hyperparameter tuning.
Start reading with the most cited papers here:
- Decoupled Weight Decay Regularization (ICLR 2019 · 9,419 citations)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (NeurIPS 2017 · 4,380 citations)
Within Optimization, its share held steady from 27.9% in 2023–24 to 29.7% in 2025–26 (242 → 290 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Optimizers in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Memory-efficient LLM optimizers rising: 5% → 14% of the topic (11 → 41 papers)
- Adam convergence and adaptive learning rates rising: 6% → 10% of the topic (15 → 30 papers)
- SGD momentum and step-size schedules rising: 3% → 7% of the topic (8 → 21 papers)
- Randomized gradient estimation falling: 11% → 3% of the topic (27 → 9 papers)
- Differentiating through embedded optimization falling: 14% → 11% of the topic (33 → 31 papers)
Papers from 2025–2026 that are already cited most:
- Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters (CVPR 2025 · 7 citations)
- Locally Convex Global Loss Network for Decision-Focused Learning (AAAI 2025 · 6 citations)
- PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization (AAAI 2025 · 6 citations)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · 4 citations)
Approaches and key papers
Differentiable losses and solvers · 469 papers
Keywords: differentiable, functions, loss
Approaches in this cluster:
- Differentiating through embedded optimization (123 papers)
Backpropagates through optimization layers and learns optimizers or proximal operators. - Surrogate losses and robust solvers (129 papers)
Proposes tailored losses, quantization regularizers, and fast solvers for specific learning problems. - Constrained and non-backprop training (108 papers)
Trains networks with orthogonality constraints, growth steps, or alternating minimization instead of plain backpropagation. - Randomized gradient estimation (79 papers)
Reduces variance in gradient estimators via randomized differentiation, evolution strategies, and weight averaging. - Multi-objective gradient methods (30 papers)
Provides stochastic gradient algorithms with convergence guarantees for multi-objective and multi-task learning.
Most cited and most cited since 2024:
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (NeurIPS 2017 · 4,380 citations)
- Deep Leakage from Gradients (NeurIPS 2019 · 1,528 citations)
- Runtime Analysis of the SMS-EMOA for Many-Objective Optimization (AAAI 2024 · 38 citations)
- LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP (CVPR 2024 · 30 citations)
Adaptive optimizers · 321 papers
Keywords: adaptive, optimizers, momentum
Approaches in this cluster:
- Adam convergence and adaptive learning rates (86 papers)
Analyzes Adam-family convergence and adapts learning rates using hypergradients. - Memory-efficient LLM optimizers (83 papers)
Designs large-batch, low-memory, and normalized optimizers for training large language models. - SGD momentum and step-size schedules (77 papers)
Studies momentum, batch size, and step-size schedules for stochastic gradient descent. - Second-order and structured preconditioning (75 papers)
Builds diagonal, full-matrix, and Hessian-based preconditioners for adaptive optimization.
Most cited and most cited since 2024:
- Decoupled Weight Decay Regularization (ICLR 2019 · 9,419 citations)
- On the Convergence of Adam and Beyond (ICLR 2018 · 1,616 citations)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · 28 citations)
- The Road Less Scheduled (NeurIPS 2024 · 16 citations)
Sampling and gradient estimators · 313 papers
Keywords: variational, sampling, inference
Approaches in this cluster:
- Stochastic gradient MCMC (157 papers)
Uses stochastic gradient Langevin dynamics and related samplers for scalable Bayesian posterior sampling. - Natural gradient and Fisher approximations (81 papers)
Develops and critiques Kronecker-factored and structured approximations of natural gradient and second-order updates. - Stein variational gradient descent (50 papers)
Transports particles with kernelized Stein variational updates for Bayesian inference, with analysis and extensions. - Zeroth-order training methods (25 papers)
Trains models using gradient estimates from function queries, with convergence and learning-dynamics analysis.
Most cited and most cited since 2024:
- Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm (NeurIPS 2016 · 307 citations)
- A Variational Inequality Perspective on Generative Adversarial Networks (ICLR 2019 · 157 citations)
- PseuZO: Pseudo-Zeroth-Order Algorithm for Training Deep Neural Networks (NeurIPS 2025 · 3 citations)
- On Estimating the Gradient of the Expected Information Gain in Bayesian Experimental Design (AAAI 2024 · 3 citations)
Bilevel and meta-learning optimization · 103 papers
Keywords: bilevel, inner, shot
Approaches in this cluster:
- Gradient-based meta-learning theory (43 papers)
Provides guarantees and improved algorithms for MAML-style outer-loop meta-learning. - Hypergradient bilevel optimization (37 papers)
Computes approximate hypergradients to solve bilevel problems for hyperparameter optimization. - Differentiating through inner loops (23 papers)
Optimizes hyperparameters over long horizons using implicit differentiation and truncated inner-loop gradients.
Most cited and most cited since 2024:
- Optimization as a Model for Few-Shot Learning (ICLR 2017 · 2,375 citations)
- Learning to Reweight Examples for Robust Deep Learning (ICML 2018 · 841 citations)
- MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning (AAAI 2024 · 68 citations)
- Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters (CVPR 2025 · 7 citations)
