Research map / Optimization / Optimizers
On the Performance Analysis of Momentum Method: A Frequency Domain Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames SGD momentum as a time-variant linear filter analyzed with the Z-transform, uses the resulting insights to design FSGDM, a heuristic optimizer with a dynamic magnitude response that beats SGDM variants on vision, NMT and RL tasks.
Problem
The optimal choice and temporal schedule of momentum coefficients in SGD with momentum is unclear, and the difference between decoupled (Standard-SGDM) and coupled (EMA-SGDM) momentum is not well understood.
Approach
Interprets the momentum update as a recursive, quasi-stationary time-variant filter on gradients, analyzes its magnitude response per stage, and derives a dynamic low-pass-gain schedule (increasing u_t, fixed v_t, scaling factor c) implemented as the FSGDM optimizer.
Key results
- FSGDM reaches 81.44% top-1 on CIFAR-100 with ResNet50 versus 79.71% for Standard-SGDM and 76.84% for EMA-SGDM.
- On ImageNet with ResNet50, FSGDM achieves 76.91% vs 76.66% for Standard-SGDM and 74.24% for EMA-SGDM.
- On IWSLT14, FSGDM improves BLEU across six models, e.g., Transformer 32.40 vs 31.50 for Standard-SGDM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FSGDM instead of Standard-SGDM | ResNet50 · CIFAR-100 | Top-1 accuracy: 79.71\textsubscript{0.25} -> 81.44\textsubscript{0.06} | Helped |
| Use Standard-SGDM instead of EMA-SGDM | CNN image classification models · Image classification benchmarks | Top-1 accuracy | Helped |
| Use FSGDM instead of Standard-SGDM | Translation models · IWSLT14 German-English | BLEU | Helped |
| Use increasing low-pass momentum instead of decreasing low-pass momentum (ablation) | ResNet50 · CIFAR-100 | Top-1 accuracy: 69.69\textsubscript{0.75} -> 77.12\textsubscript{0.07} | Helped |
| Use high-pass momentum with decreasing u_t instead of low-pass momentum with the same decreasing factor (ablation) | ResNet50 · CIFAR-100 | Top-1 accuracy: 69.69\textsubscript{0.75} -> 70.99\textsubscript{0.18} | Helped |
| Use increasing low-pass gain momentum instead of fixed low-pass gain momentum (ablation) | ResNet50 · CIFAR-100 | Top-1 accuracy: 79.71\textsubscript{0.25}; 80.48\textsubscript{0.03} | Mixed |
| Replace the FSGDM sequence with linear, exponential, sine, or logarithmic sequences (ablation) | FSGDM with ResNet50 · CIFAR-100 | Top-1 accuracy: 81.44\textsubscript{0.06} -> 78.24\textsubscript{0.24}; 80.38\textsubscript{0.04}; 78.76\textsubscript{0.29}; 78.70\text | Hurt |
| Replace FSGDM with LP2HP, HP2LP, LPG2HPG, or HPG2LPG (ablation) | ResNet50 · CIFAR-100 | Top-1 accuracy: 81.44\textsubscript{0.06} -> 74.77\textsubscript{0.21}; 77.00\textsubscript{0.13}; 72.60\textsubscript{0.58}; 78.91\text | Hurt |
Surprising findings
- Coupled EMA-SGDM with u=0.9 underperforms decoupled Standard-SGDM substantially on NLP tasks (e.g., LSTM-W 1.20 vs 24.66 BLEU).
- High-pass momentum with decreasing coefficients can outperform low-pass in some settings, suggesting high-frequency gradient components matter early in training.
Limitations
- FSGDM is a heuristic; the optimal filter strategy is not derived and hyperparameters (c, v) were tuned empirically on a limited set of tasks.
- Frequency analysis does not extend to adaptive optimizers like Adam/RMSprop due to nonlinearity of second-moment estimates.
- Hyperparameters were shared across models rather than fine-tuned per model, and results average only 3 seeds.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet IWSLT14 German-English
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet IWSLT14 German-English MuJoCo Walker2d-v4 HalfCheetah-v4 Ant-v4
Metrics: top-1 accuracy BLEU episode reward
Baselines: Standard-SGDM EMA-SGDM Adam
Builds on: SGD with momentum exponential moving average momentum Adam PPO
Closest papers by meaning
- The Marginal Value of Momentum for Small Learning Rate SGD (ICLR 2024 · similarity 0.90)
- Random Scaling and Momentum for Non-smooth Non-convex Optimization (ICML 2024 · similarity 0.88)
- A geometric framework for momentum-based optimizers for low-rank training (NeurIPS 2025 · similarity 0.87)
- Gradient correlation is a key ingredient to accelerate SGD with momentum (ICLR 2025 · similarity 0.87)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.86)
- Dimension-adapted Momentum Outscales SGD (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.