NeurIPS 2024 accepted papers on Optimization

191 of the 4,560 papers accepted at NeurIPS 2024 (4.2%) are on Optimization, compared with 4.9% at NeurIPS 2023. They fall into 4 topics.

See NeurIPS 2024 in the interactive map

Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.

Topics and papers

Nonconvex and smooth optimization · 71 papers

Adaptive Proximal Gradient Method for Convex Optimization (13 citations)
Adam with model exponential moving average is effective for nonconvex optimization (9 citations)
In-and-Out: Algorithmic Diffusion for Sampling Convex Bodies (4 citations)
Fundamental Convergence Analysis of Sharpness-Aware Minimization (3 citations)
On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (3 citations)
On the Stability and Generalization of Meta-Learning (3 citations)
Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptions (2 citations)
First-Order Methods for Linearly Constrained Bilevel Optimization (2 citations)
Non-asymptotic Global Convergence Analysis of BFGS with the Armijo-Wolfe Line Search (2 citations)
Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction (2 citations)
A Framework for Bilevel Optimization on Riemannian Manifolds (1 citation)
A Primal-Dual-Assisted Penalty Approach to Bilevel Optimization with Coupled Constraints (1 citation)

and 59 more in the interactive map.

Optimizers · 54 papers

The Road Less Scheduled (16 citations)
Dual Cone Gradient Descent for Training Physics-Informed Neural Networks (10 citations)
Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement (8 citations)
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning (7 citations)
Why Transformers Need Adam: A Hessian Perspective (7 citations)
BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning (5 citations)
Certified Machine Unlearning via Noisy Stochastic Gradient Descent (5 citations)
Gradient Guidance for Diffusion Models: An Optimization Perspective (4 citations)
Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling (4 citations)
Loss Landscape Characterization of Neural Networks without Over-Parametrization (3 citations)
Memory-Efficient LLM Training with Online Subspace Descent (3 citations)
Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization (3 citations)

and 42 more in the interactive map.

Implicit bias and generalization · 52 papers

Why Warmup the Learning Rate? Underlying Mechanisms and Improvements (15 citations)
Why Do We Need Weight Decay in Modern Deep Learning? (10 citations)
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models (6 citations)
Identifying Equivalent Training Dynamics (6 citations)
How does PDE order affect the convergence of PINNs? (5 citations)
Implicit Optimization Bias of Next-token Prediction in Linear Models (2 citations)
Neglected Hessian component explains mysteries in sharpness regularization (2 citations)
On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (2 citations)
4+3 Phases of Compute-Optimal Neural Scaling Laws (1 citation)
A generalized neural tangent kernel for surrogate gradient learning (1 citation)
Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training (1 citation)
Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (1 citation)

and 40 more in the interactive map.

Distributed and compressed · 14 papers

and 2 more in the interactive map.