AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

On the Generalization of Stochastic Gradient Descent with Momentum

Ali Ramezani-Kebrya, Kimon Antonakopoulos, Volkan Cevher, Ashish Khisti, Ben Liang

JMLR 2024 · Optimization › Implicit bias and generalization · approach: SGD noise and flat minima

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add momentum only for the early phase of training (SGDEM) instead of no momentum (ablation)SGD · notMNIST (tuned, td=3000)test error: 0.3596 ± 0.0142 -> 0.2892 ± 0.0042Improved
restrict momentum to the first td iterations (SGDEM) instead of the entire training (SGDM) (ablation)SGDM · notMNIST (tuned, td=3000)test error: 0.3194 ± 0.0030 -> 0.2892 ± 0.0042Improved
add momentum for the entire training (SGDM) instead of early only, for large momentum 0.99 on nonconvex loss (ablation)SGDEM · notMNIST nonconvextest errorHurt
increase momentum parameter mu in strongly convex logistic regression (ablation)SGD · notMNIST logistic regression (generalization error)generalization errorHurt
add momentum (mu 0.5 or 0.9) to SGD in strongly convex logistic regression (ablation)SGD · notMNIST logistic regression (training error)training errorImproved
add momentum to SGD for many epochs in logistic regression (ablation)SGD · notMNIST logistic regression (n=500)test accuracyNo clear effect
apply momentum 0.9 for the first 50 epochs only instead of all 90 epochs (ablation)SGDM · ImageNet ResNet-18generalization errorImproved
choose intermediate td for early momentum, small mu_d (ablation)SGDEM · CIFAR10 ResNet-20test errorImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy