Research map / Optimization / Implicit bias and generalization
Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes how batch size and weight decay affect generalization of stochastic Adam and AdamW on two-layer CNNs with feature-noise data, showing large-batch variants overfit noise while mini-batch variants generalize, and Adam tolerates a much smaller weight decay than AdamW.
Problem
Full-batch analyses of Adam do not capture stochastic (mini-batch) Adam, which does not converge to its full-batch counterpart even with infinitesimal learning rates; how batch size and weight decay affect Adam/AdamW generalization is unclear.
Approach
Extends the feature-noise two-layer CNN analysis of full-batch Adam to stochastic Adam and AdamW, showing a batch-size threshold (n/B) separates noise-overfitting large-batch solutions from near-zero-test-error mini-batch solutions, and that Adam's effective weight decay bound (sigma_0^{q-2}) is much smaller than AdamW's (B^2/n ∧ 1 to O(1)); mini-batch Adam is approximated by SignSGD.
Key results
- Large-batch Adam and AdamW (n/B = Theta(1)) provably reach zero training error but test error at least 1/2 - o(1) under weight decay.
- Mini-batch Adam and AdamW (n/B >= Theta(log 1/epsilon)) achieve zero training error and test error o(1) in theory.
- Adam's admissible effective weight decay is bounded by O(sigma_0^{q-2}) while AdamW tolerates weight decay from roughly B^2/n ∧ 1 up to O(1); experiments show Adam error spikes for lambda > 0.05 while AdamW stays robust up to 0.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mini-batch rather than large-batch training | Adam · Synthetic image data model | test error: ≥ 1/2-o(1) -> o(1) | Helped |
| Use mini-batch rather than large-batch training | AdamW · Synthetic image data model | test error: ≥ 1/2-o(1) -> o(1) | Helped |
| Increase batch size beyond a critical threshold | Adam and AdamW · CIFAR-10 | test error | Hurt |
| Set Adam weight decay above 0.05 | Adam · CIFAR-10, batch size 16 | test error | Hurt |
| Increase AdamW weight decay up to 0.5 | AdamW · CIFAR-10, batch size 16 | test error | No clear effect |
Surprising findings
- Stochastic Adam with small batch sizes generalizes well while full-batch Adam with the same weight decay fails, despite identical behavior in the infinitesimal-learning-rate GD limit.
- Adam requires a weight decay several orders of magnitude smaller than AdamW to avoid training stagnation.
Limitations
- Analysis restricted to two-layer CNNs on a simplified single-feature, single-noise-patch data model.
- Does not consider momentum, learning rate schedules, or gradient clipping.
- Only image data; NLP and deeper architectures not analyzed.
Experimental setup
Datasets: CIFAR-10 ImageNet-1K synthetic feature-noise data
Benchmarks: CIFAR-10 ImageNet-1K
Metrics: test error training error
Baselines: full-batch Adam full-batch AdamW SGD GD
Builds on: Adam AdamW SignSGD feature learning theory of full-batch Adam vs GD
Closest papers by meaning
- Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization (ICLR 2023 · similarity 0.93)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.91)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.89)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.89)
- Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling (NeurIPS 2024 · similarity 0.89)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.