Research map / Optimization / Optimizers
The Marginal Value of Momentum for Small Learning Rate SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that SGD with momentum (SGDM) and vanilla SGD with learning rate γ/(1-β) track each other in small-learning-rate, noise-dominated regimes, and empirically that momentum gives limited benefit on ImageNet, RoBERTa fine-tuning, and CIFAR-10 with SVAG.
Problem
Whether momentum provably helps stochastic gradient optimization in deep learning, where gradient noise dominates and learning rates are small; prior analyses found no provable acceleration of SGDM over SGD.
Approach
Uses weak approximation theory and an auxiliary coupled trajectory to show SGDM is weakly close to SGD (with averaged learning rate) over O(1/η) steps under noise scaling σ≤η^{-1/2}, and to share the same slow SDE on the manifold of minimizers over O(1/η²) steps; validates with ImageNet, RoBERTa fine-tuning, and SVAG-modified CIFAR-10 experiments.
Key results
- On RoBERTa-large few-shot fine-tuning, SGD and SGDM are close across 5 tasks (e.g., SST-2 94.0 vs 94.0, SNLI 87.7 vs 88.4).
- On ImageNet ResNet-50, SGDM matches SGD at batch size 1024 but outperforms SGD significantly at larger batch sizes.
- On CIFAR-10 ResNet-32 with batch 512, applying SVAG with increasing ℓ closes the gap between SGDM and SGD, leaving no evident SGDM edge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add momentum (SGD to SGDM) | RoBERTa-large fine-tuning · SST-2 | accuracy: 94.0 (0.4) -> 94.0 (0.5) | No clear effect |
| Add momentum (SGD to SGDM) | RoBERTa-large fine-tuning · SST-5 | accuracy: 55.2 (1.1) -> 55.0 (1.0) | No clear effect |
| Add momentum (SGD to SGDM) | RoBERTa-large fine-tuning · SNLI | accuracy: 87.7 (0.3) -> 88.4 (0.6) | Mixed |
| Add momentum (SGD to SGDM) | RoBERTa-large fine-tuning · TREC | accuracy: 97.2 (0.2) -> 97.2 (0.4) | No clear effect |
| Add momentum (SGD to SGDM) | RoBERTa-large fine-tuning · MNLI | accuracy: 84.0 (0.3) -> 83.7 (0.8) | Mixed |
| Use larger SVAG noise-scale multiplier ℓ (ablation) | SGD and SGDM on ResNet-32 · CIFAR-10, batch size B=512 | test performance | No clear effect |
Surprising findings
- Momentum's variance reduction is offset by long-range correlation, so SGDM trajectories with the same learning rate have the same distributional limit as SGD despite smoother single-step updates.
- Reducing curvature-induced effects via SVAG eliminates SGDM's advantage over SGD, suggesting momentum's benefit comes mainly from curvature rather than noise reduction.
Limitations
- Theory relies on asymptotic small learning rate and bounded/smooth assumptions; weak approximation bound O(sqrt(η/(1-β))) can be loose at large learning rates.
- Large-batch ImageNet experiments show SGDM benefits when optimal learning rate is large, where the theory does not apply.
Experimental setup
Datasets: ImageNet CIFAR-10 SST-2 SST-5 SNLI TREC MNLI
Benchmarks: ImageNet CIFAR-10 SST-2 SST-5 SNLI TREC MNLI
Metrics: test accuracy
Baselines: SGD SGDM SVAG
Builds on: SVAG Linear Scaling Rule Li et al. 2021 slow SDE analysis of SGD Polyak heavy-ball momentum
Closest papers by meaning
- Towards understanding how momentum improves generalization in deep learning (ICML 2022 · similarity 0.92)
- Gradient correlation is a key ingredient to accelerate SGD with momentum (ICLR 2025 · similarity 0.91)
- Random Scaling and Momentum for Non-smooth Non-convex Optimization (ICML 2024 · similarity 0.90)
- On the Performance Analysis of Momentum Method: A Frequency Domain Perspective (ICLR 2025 · similarity 0.90)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.90)
- Dimension-adapted Momentum Outscales SGD (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.