Research map / Optimization / Nonconvex and smooth optimization
Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces (L, L̄)-anisotropic smoothness via abstract convexity (Φ-convexity) and analyzes nonlinearly preconditioned gradient descent, which recovers gradient clipping, Adam-like and Adagrad-like updates, with nonconvex and convex convergence results.
Problem
Smooth minimization beyond Lipschitz smoothness; unifying clipping, signed/adaptive gradient methods and (L0,L1)-smooth settings under a majorization-minimization view with nonlinear gradient preconditioning.
Approach
Generalizes anisotropic smoothness to a reference function that may be nonsmooth or have non-full domain and two constants (L, L̄); gives first- and second-order sufficient conditions via the forward operator being a homeomorphism, and shows the preconditioned iteration with stepsizes up to 2/L converges in nonconvex settings and at O(1/K) in convex settings.
Key results
- Nonconvex setting: stepsizes gamma = alpha/L with alpha in (0,2) are allowed, giving min_k phi(grad phi*(grad f/L̄)) ≤ L(f(x0)-f*)/(L̄ β(K+1)) with β = 1-|1-α|.
- Convex isotropic case with gamma=1/L and h' (x)/x decreasing: f(x^K)-f* ≤ L||∇f(x0)|| ||x0-x*||^2 /(h*'(||∇f(x0)/L̄||)(K+1)), an O(1/K) rate.
- Convex anisotropic case with 2-subhomogeneous reference function: f(x^K)-f* ≤ 4D0/K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed isotropic or anisotropic preconditioned method generated by cosh−1 instead of vanilla gradient descent, gradient clipping, or the tuned comparison method | Nonconvex phase retrieval · Nonconvex phase retrieval | cost function value | Helped |
| use the isotropic proposed algorithm instead of gradient clipping with the tested tuning | Gradient clipping · Nonconvex phase retrieval, 100 random instances | cost function value | Helped |
| vary L and L̄ for the proposed preconditioned methods | Proposed preconditioned methods using cosh, exponential, or logarithmic reference functions · Norm to power, f(x)=¼‖x‖⁴ | convergence speed | Mixed |
| vary γ and λ | Proposed methods and gradient clipping · Simple neural network training on MNIST subset | training behavior | Mixed |
Surprising findings
- The forward-step algorithm admits stepsizes up to 2/L in the nonconvex smooth setting, improving on the usual 1/L restriction.
Limitations
- Analysis is for full-gradient (deterministic) setting without momentum.
- Convex rate for isotropic case depends on the initial gradient norm; anisotropic O(1/K) rate requires subhomogeneous reference and bounded initial level sets.
Experimental setup
Datasets: MNIST
Metrics: gradient norm function value suboptimality
Baselines: gradient descent gradient clipping Adagrad Adam
Builds on: anisotropic smoothness dual relative smoothness (L0,L1)-smoothness gradient clipping
Closest papers by meaning
- Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis (NeurIPS 2025 · similarity 0.93)
- Escaping saddle points without Lipschitz smoothness: the power of nonlinear preconditioning (NeurIPS 2025 · similarity 0.92)
- Efficiently Escaping Saddle Points under Generalized Smoothness via Self-Bounding Regularity (NeurIPS 2025 · similarity 0.91)
- Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity (ICLR 2025 · similarity 0.91)
- Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods (ICLR 2025 · similarity 0.91)
- Toward a Unified Theory of Gradient Descent under Generalized Smoothness (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.