Research map / Optimization / Nonconvex and smooth optimization
Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a heavy-ball momentum variant and a stochastic extension of nonlinearly preconditioned (anisotropic) gradient descent with sigmoid-type preconditioners, with convergence guarantees under anisotropic smoothness and tests on MNIST, CIFAR-10 and matrix factorization.
Problem
Nonconvex smooth optimization under generalized smoothness beyond Lipschitz gradients (e.g., (L0,L1)-smoothness), where standard GD/SGD require careful tuning; existing preconditioned/clipping methods lack momentum analysis and general stochastic analysis.
Approach
Apply heavy-ball momentum to the preconditioned gradient (convex combination of nabla phi*(grad f) terms) and analyze it using only convexity of the reference function under anisotropic descent inequality; extend to a stochastic version with a noise condition weaker than bounded variance.
Key results
- Momentum variant iHGDm significantly outperforms GD, GDm and AdGD-accel on matrix factorization across ranks r=10, 20, 30 and is more stable across random initializations.
- On MNIST MLP, iHGD reaches small training losses faster than SGD and Adam.
- On CIFAR-10 ResNet-18, proposed methods perform similarly to SGD and SGDm, which are the state of the art for this problem.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iHGD instead of SGD or Adam | MNIST MLP · MNIST | training loss | Helped |
| Use the proposed methods instead of SGD and SGDm | ResNet-18 · CIFAR-10 | test accuracy | No clear effect |
| Use iHGDm instead of GD, GDm, or AdGD-accel | Matrix factorization · MovieLens 100K, r=10, 20, 30 | training loss | Helped |
Surprising findings
- The stochastic noise condition in the paper is shown on an example to hold where per-sample gradient variance is unbounded, so it can be less restrictive than bounded variance.
Limitations
- Momentum analysis only holds for beta in [0,0.5) in one theorem and (0,1) with a different stepsize in another; general beta in [0,1) remains open.
- No stochastic analysis for the momentum method; stochastic analysis for batch-size-dependent noise requires large batches.
Experimental setup
Datasets: MNIST CIFAR-10 MovieLens 100K
Benchmarks: MNIST CIFAR-10 MovieLens 100K
Metrics: training loss test accuracy objective value
Baselines: SGD SGD with momentum Adam gradient descent gradient descent with momentum AdGD-accel
Builds on: anisotropic descent inequality (L0,L1)-smoothness gradient clipping heavy ball method Adam
Closest papers by meaning
- Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness (ICML 2025 · similarity 0.93)
- Escaping saddle points without Lipschitz smoothness: the power of nonlinear preconditioning (NeurIPS 2025 · similarity 0.91)
- Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions (ICML 2024 · similarity 0.89)
- Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity (ICLR 2025 · similarity 0.88)
- Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods (ICLR 2025 · similarity 0.88)
- Accelerated Over-Relaxation Heavy-Ball Method: Achieving Global Accelerated Convergence with Broad Generalization (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.