Research map / Optimization / Implicit bias and generalization
Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase basin Radon measure (basin volume) | SGD SDE · Toy basins A, B, and C | average iterations for successful escaping: 122 -> 457 -> 1898 | Hurt |
| Increase basin Radon measure (basin volume) | SGD SDE · Toy basins A, B, and C | escaping probability: 100% -> 65.6% -> 10.1% | Hurt |
| Use anisotropic gradient noise without adaptive coordinate scaling instead of Adam-style geometry adaptation | SGD versus Adam · Local basin escaping | escaping time | Helped |
| Smooth gradient noise with an exponential moving average | Adam versus SGD · Local basin escaping | escaping time | Hurt |
| Use heavy-tailed Levy noise rather than Gaussian noise to model gradient noise | SGD and Adam SDEs · AlexNet on CIFAR10 | gradient-noise fit | Helped |
| Use SGD rather than Adam | Fully connected networks · MNIST and CIFAR10 | test accuracy | Helped |
| Use SGD rather than Adam | Fully connected networks · MNIST and CIFAR10 | basin-transition behavior | Mixed |
Closest papers by meaning
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.90)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.89)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.89)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.89)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.89)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.