Research map / Optimization / Implicit bias and generalization

Investigating the Role of Weight Decay in Enhancing Nonconvex SGD

CVPR 2025 · Optimization › Implicit bias and generalization · approach: SGD noise and flat minima · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives convergence and uniform-stability generalization theory for SGD with decoupled weight decay (SGDW) and SignSGD with momentum on nonconvex objectives, showing weight decay does not speed convergence but improves generalization when δ<1, with CIFAR/ImageNet experiments.

Problem

Lack of theoretical understanding of why SGD with weight decay yields results consistent with the unregularized model and how weight decay improves generalization in nonconvex optimization.

Approach

Analyzes SGDW on the original (unregularized) nonconvex objective under weaker assumptions, introduces a gradient-weight alignment quantity δT, and shows via algorithmic stability that a large decay rate (1-α ~ γ) tightens the generalization bound, extending the analysis to SignSGD with momentum.

Key results

• SGDW convergence slows as weight decay increases; wd=0 converges fastest across VGG16/ResNet20/ResNet18 experiments.
• Except for wd=5e-2 on VGG16 CIFAR-10 (did not converge), SGDW with moderate weight decay outperforms SGD without weight decay in test accuracy and loss.
• SignSGDW converges and generalizes at larger weight decay (5e-2) where SGDW fails, but SGDW generalizes better than SignSGDW in the reported comparison.

Measured effects reported in the paper

Surprising findings

• Weight decay does not accelerate SGD convergence, contrary to the common intuition that it helps optimization.
• SignSGDW is more robust to large weight decay than SGDW, but still generalizes worse than SGDW.

Limitations

• Generalization benefit holds only when delta_T is well below 1 (gradient and weights not nearly linearly dependent), which is an assumption rather than guaranteed.
• Small-decay-rate analysis yields no clear advantage for weight decay in convergence or generalization.
• Experiments limited to image classification with standard CNNs (VGG, ResNet).

Experimental setup

Datasets: CIFAR-10 CIFAR-100 ImageNet

Benchmarks: CIFAR-10 CIFAR-100 ImageNet

Metrics: training loss test loss test accuracy gradient norm delta

Baselines: SGD without weight decay SignSGDW SGDW

Builds on: SGDW (decoupled weight decay) uniform stability for nonconvex SGD SignSGD with momentum AdamW

Closest papers by meaning

• Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions (NeurIPS 2026 · similarity 0.90)
• Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.90)
• On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective (NeurIPS 2023 · similarity 0.90)
• Decoupled Weight Decay Regularization (ICLR 2019 · similarity 0.90)
• Asymptotic theory of SGD with a general learning-rate (NeurIPS 2025 · similarity 0.89)
• When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds (NeurIPS 2026 · similarity 0.88)