Research map / Optimization / Implicit bias and generalization
A Bayesian Perspective on Generalization and Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train logistic regression on random rather than informative labels | Logistic regression · MNIST | test accuracy | Hurt |
| Increase regularization from weak to an intermediate strength | Logistic regression · MNIST with informative labels | test cross-entropy | Helped |
| Increase regularization beyond the strength that gives the lowest test cross-entropy | Logistic regression · MNIST with informative labels | test cross-entropy | Hurt |
| Use small-batch SGD instead of full-batch training | Shallow neural network with 800 hidden units · MNIST | test accuracy | Helped |
| Increase batch size while holding other SGD hyperparameters fixed | Shallow neural network with 800 hidden units · MNIST | test accuracy | Mixed |
| Add L2 regularization (lambda = 0.1) (ablation) | Shallow neural network with small-batch and full-batch SGD · MNIST | generalization gap in test accuracy | Helped |
| Add L2 regularization (lambda = 0.1) (ablation) | Shallow neural network with small-batch and full-batch SGD · MNIST | test cross-entropy | Helped |
| Increase learning rate while increasing batch size to preserve the noise scale | SGD · MNIST | test accuracy | Helped |
| Increase learning rate beyond approximately 3 | SGD · MNIST | peak test accuracy | Hurt |
| Increase training set size while scaling batch size | SGD · MNIST | test accuracy | Helped |
| Increase momentum coefficient and scale batch size accordingly | SGD with momentum · MNIST | test accuracy | No clear effect |
Closest papers by meaning
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (ICML 2020 · similarity 0.94)
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.93)
- Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence (NeurIPS 2019 · similarity 0.91)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.90)
- Strength of Minibatch Noise in SGD (ICLR 2022 · similarity 0.89)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.