Research map

Optimization: research map

Convergence, optimizers, distributed training and implicit bias.

3,879 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026), in 4 topics. Within all five venues, its share shrank from 3.3% in 2023–24 to 2.2% in 2025–26 (868 → 978 papers at ICML, NeurIPS, CVPR and AAAI, the venues with data for all four years).

Explore Optimization in the interactive map

Topics

• Nonconvex and smooth optimization · 1,448 papers
  Adaptive first-order convex methods, Stochastic gradient last-iterate analysis, Coordinate and Adam convergence analysis
• Optimizers · 1,124 papers
  Differentiating through embedded optimization, Surrogate losses and robust solvers, Constrained and non-backprop training
• Implicit bias and generalization · 958 papers
  Shallow ReLU network convergence, Neural tangent kernel analysis, Gradient flow and mean-field dynamics
• Distributed and compressed · 349 papers
  Decentralized optimization algorithms, Local and asynchronous SGD, Byzantine-robust learning

Most cited papers in Optimization

• Decoupled Weight Decay Regularization (ICLR 2019 · 9,419 citations)
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (NeurIPS 2017 · 4,380 citations)
• Optimization as a Model for Few-Shot Learning (ICLR 2017 · 2,375 citations)
• SGDR: Stochastic Gradient Descent with Warm Restarts (ICLR 2017 · 1,699 citations)
• On the Convergence of Adam and Beyond (ICLR 2018 · 1,616 citations)
• Deep Leakage from Gradients (NeurIPS 2019 · 1,528 citations)
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · 1,345 citations)
• Understanding deep learning requires rethinking generalization (ICLR 2017 · 1,040 citations)
• On the Convergence of FedAvg on Non-IID Data (ICLR 2020 · 979 citations)
• Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent (NeurIPS 2017 · 967 citations)
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · 911 citations)
• Learning to Reweight Examples for Robust Deep Learning (ICML 2018 · 841 citations)