Research map / Optimization / Nonconvex and smooth optimization

Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness

NeurIPS 2025 · Optimization › Nonconvex and smooth optimization · approach: Projection-free conditional gradient · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes generalized gradient norm clipping (GGNC) and a stochastic short-step Frank-Wolfe variant (S³CG) for non-Euclidean optimization, analyzed under a non-Euclidean (L0,L1)-smoothness assumption, and instantiates them as Clipped Scion for deep learning.

Problem

Gradient norm clipping and conditional gradient methods each have complementary convergence properties; existing analyses of clipping are Euclidean and lack stochastic guarantees, and weight decay is not principled for clipped LMO-based methods.

Approach

A hybrid of steepest descent (sharp operator) and LMO-based conditional gradient with a clipped step size min{1, rho/||d||_*}, which generalizes Euclidean clipping to arbitrary norms and is shown to be descent under a generalized (L0,L1)-smoothness; weight decay is tied to the Frank-Wolfe short step.

Key results

• Deterministic GGNC attains min gradient-norm bound sqrt(L0*Delta/n)+2L1*Delta/n with gamma=1/L0, rho=L0/L1.
• Stochastic GGNC with momentum achieves O(n^{-1/4}) expected gradient-norm rate under (L0,L1)-smoothness and L-smoothness with alpha=1/sqrt(n).
• S³CG (stochastic short-step Frank-Wolfe) achieves O(n^{-1/4}) rate for the Wolfe gap, described as the first stochastic short-step convergence proof.

Limitations

• Stochastic rate is O(n^{-1/4}) with horizon-dependent parameter choices, extended to anytime only by remark.
• Non-descent for the unclipped conditional gradient method requires diminishing or constrained stepsizes.
• Stochastic convergence for the stochastic case relies on bounded variance and an L-smoothness assumption on the constrained set.

Experimental setup

Baselines: Scion steepest descent conditional gradient (uCG)

Builds on: Scion gradient norm clipping (L0,L1)-smoothness Frank-Wolfe short step normalized SGD (LARS)

Closest papers by meaning

• Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity (ICLR 2025 · similarity 0.90)
• Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions (NeurIPS 2025 · similarity 0.89)
• Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.88)
• Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization (ICML 2025 · similarity 0.88)
• Parameter-free Clipped Gradient Descent Meets Polyak (NeurIPS 2024 · similarity 0.87)
• To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.87)