Research map / Optimization / Nonconvex and smooth optimization

Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions

NeurIPS 2026 · Optimization › Nonconvex and smooth optimization · approach: Stochastic gradient last-iterate analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically that learning rate decay in SGD can exponentially accelerate global optimization of nonconvex functions compared with constant step sizes.

Problem

Whether learning rate decay offers an asymptotic advantage for global optimization of nonconvex functions with SGD, which is unclear despite practitioner belief that decay helps.

Approach

Demonstrates a natural class of nonconvex functions where SGD with learning rate decay converges exponentially faster for global optimization than constant-learning-rate SGD.

Key results

• Abstract claims an affirmative answer: a natural class of nonconvex functions where learning rate decay yields exponential acceleration of SGD for global optimization (exact rates not stated in abstract).

Experimental setup

Baselines: SGD with constant learning rate

Builds on: stochastic gradient descent

Closest papers by meaning

• Asymptotic theory of SGD with a general learning-rate (NeurIPS 2025 · similarity 0.92)
• Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · similarity 0.90)
• The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.89)
• General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization (ICML 2025 · similarity 0.88)
• Rapid Overfitting of Multi-Pass SGD in Stochastic Convex Optimization (ICML 2025 · similarity 0.88)
• Statistical Guarantees for High-Dimensional Stochastic Gradient Descent (NeurIPS 2025 · similarity 0.88)