Research map / Optimization / Implicit bias and generalization
On the Noisy Gradient Descent that Generalizes as SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SGD sampling noise with Gaussian sampling noise matched to the SGD covariance | Gradient descent (MSGD-Cov) · Various datasets and models | generalization | No clear effect |
| Use Gaussian sampling noise matched to the SGD covariance | Gradient descent (MSGD-Cov) · Various datasets and models | convergence speed | Helped |
| Use scaled-Fisher Gaussian noise instead of Gaussian noise with SGD covariance | MSGD-Cov · Various datasets and models | generalization | No clear effect |
| Use independent-component Bernoulli sampling noise with the diagonal of the SGD sampling covariance | Gradient descent (MSGD-Bernoulli) · Various datasets and models | generalization | No clear effect |
| Use sparse Gaussian sampling noise based on a mini-batch Fisher estimate | Gradient descent (MSGD-[Fisher-B]) · FashionMNIST and SVHN | generalization | No clear effect |
| Add compensatory sparse Gaussian sampling noise to large-batch training | Mini-batch MSGD ([MSGD-Fisher]-B) · Toy datasets | generalization | Helped |
| Add compensatory sparse Gaussian sampling noise to large-batch training | Mini-batch MSGD ([MSGD-Fisher]-B) · ResNet-18 on CIFAR10 | generalization | Helped |
| Use Gaussian MSGD rather than other tested noise variants | MSGD · Empirical studies across datasets and models | convergence speed | Helped |
| Preserve the full gradient matrix while using independent-component sampling noise, rather than using independent-component gradient noise | Noisy gradient descent · Various datasets and models | generalization | Helped |
| Use large-batch MSGD with sampling-vector covariance matched to small-batch SGD | Online linear regression · Quadratic loss | generalization error | No clear effect |
Closest papers by meaning
- Strength of Minibatch Noise in SGD (ICLR 2022 · similarity 0.93)
- Label Noise SGD Provably Prefers Flat Global Minimizers (NeurIPS 2021 · similarity 0.91)
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (ICML 2020 · similarity 0.91)
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.91)
- Noise against noise: stochastic label noise helps combat inherent label noise (ICLR 2021 · similarity 0.90)
- Anticorrelated Noise Injection for Improved Generalization (ICML 2022 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.