Research map / Optimization / Nonconvex and smooth optimization

Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

NeurIPS 2026 · Optimization › Nonconvex and smooth optimization · approach: Coordinate and Adam convergence analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves high-probability and expectation convergence of Adam under L0-Lp generalized smoothness and generalized ABC second-moment noise, using self-normalized stopping-time localization, with a matching hard instance showing the delta^{-1/2} confidence dependence is sharp.

Problem

Convergence of Adam on generalized smooth objectives when stochastic gradients have only second-moment (not tail) control, with high-probability and expected stationarity guarantees.

Approach

Extends the self-normalized stopping-time analysis of Adam (developed for classical smoothness and bounded variance) to L0-Lp smoothness and generalized second-moment ABC noise, with an augmented energy and stopped localization; de-preconditioning incurs a delta^{-1/2} factor, shown sharp by a hard instance.

Key results

• High-probability bound (1/T) sum ||grad f(theta_t)||^2 = O~(d/sqrt(delta T) + d^2/T) with probability at least 1-delta, for p<2 under second-moment noise.
• Expected bound O(d(1+log T)/sqrt T + d^2(1+log T)^2/T) for p<1.
• Hard one-dimensional instance shows the delta^{-1/2} confidence dependence is sharp under second-moment information; for 1<=p<2 expectation lower bound Omega(T^{-1/3}) for the calibrated Adam family.

Surprising findings

• For 1<=p<2 the polylog(T)/sqrt(T) expectation rate fails uniformly over the full generalized ABC class for the calibrated Adam family, with an Omega(T^{-1/3}) lower bound.
• Under only second-moment information the confidence dependence delta^{-1/2} cannot in general be replaced by a polylogarithmic one.

Limitations

• The delta^{-1/2} factor in the high-probability bound is sharp for second-moment noise, so tail assumptions are needed to remove it.
• Results rely on horizon-dependent calibration beta2=1-1/T and alpha proportional to 1/sqrt(T), with constants not optimized.
• Expectation rate for 1<=p<2 fails uniformly for the calibrated Adam family, so only p<1 gets the expectation rate.

Experimental setup

Metrics: average squared gradient norm high-probability stationarity bound expected stationarity bound

Baselines: Adam under bounded or sub-Gaussian noise analyses AdamW expected l1 rate analysis

Builds on: Adam L0-Lp smoothness self-normalized Adam analysis

Closest papers by meaning

• On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.91)
• A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
• Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
• The Implicit Bias of Adam on Separable Data (NeurIPS 2024 · similarity 0.86)
• Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.86)
• Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.86)