Research map / Optimization / Nonconvex and smooth optimization
Understanding the Role of Momentum in Stochastic Gradient Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QHM with diminishing learning rates and β_k → 0 | QHM · Smooth nonconvex stochastic optimization | almost-sure convergence to a stationary point | Helped |
| Increase momentum so ν_kβ_k → 1 sufficiently slowly | QHM · Smooth nonconvex stochastic optimization with bounded noise | almost-sure convergence to a stationary point | Helped |
| Increase ν while re-optimizing α and β | QHM · Strictly convex quadratic functions | local convergence rate | Helped |
| Decrease the learning rate α | QHM · Quadratic, MNIST logistic regression, and CIFAR-10 ResNet-18 experiments | stationary mean loss | Helped |
| Increase β toward 1 | QHM · Quadratic, MNIST logistic regression, and CIFAR-10 ResNet-18 experiments | stationary mean loss | Helped |
| Increase ν | QHM · Stationary distribution on quadratic problems | stationary mean loss | Mixed |
| For large β, use an intermediate ν rather than fixing ν = 1 | QHM · Quadratic, MNIST logistic regression, and CIFAR-10 ResNet-18 experiments | stationary mean loss | Helped |
| Decrease α within the stable region rather than choosing the largest convergent α | Stochastic Heavy Ball (ν = 1) · Constant-and-drop training; quadratic problem and CIFAR-10 ResNet-18 | stationary mean loss | Helped |
| Reduce α while the algorithm remains in the regime where the convergence rate is independent of α | Stochastic Heavy Ball (ν = 1) · Stable local dynamics | local convergence rate | No clear effect |
| Use larger β | Normalized QHM · Quadratic objective with constant parameters, first-order expansion in α | stationary covariance | No clear effect |
Closest papers by meaning
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.90)
- The Marginal Value of Momentum for Small Learning Rate SGD (ICLR 2024 · similarity 0.89)
- On the Performance Analysis of Momentum Method: A Frequency Domain Perspective (ICLR 2025 · similarity 0.89)
- Gradient correlation is a key ingredient to accelerate SGD with momentum (ICLR 2025 · similarity 0.89)
- Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise (ICLR 2024 · similarity 0.89)
- Dimension-adapted Momentum Outscales SGD (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.