Research map / Optimization / Implicit bias and generalization
Strength of Minibatch Noise in SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rescale the learning rate by 1/d as network width increases | Two-layer tanh network trained with SGD · MNIST | variance of training loss | Helped |
| Rescale the learning rate by 1/d as network width increases | Two-layer tanh network trained with Adam · MNIST | training stability | Helped |
| Use a large learning rate | Linear regression with L2 regularization · Test loss | test loss | Mixed |
| Allow negative weight decay to offset the implicit regularization from a large learning rate | Logistic regressor trained with SGD · MNIST | test performance | Helped |
| Use the exact discrete-time minibatch noise model instead of the Hessian approximation | SGD fluctuation predictions · Linear regression experiments | accuracy of fluctuation prediction | Helped |
| Use the continuous-time Hessian approximation instead of the exact discrete-time minibatch noise model | SGD fluctuation predictions · Linear regression experiments | accuracy of fluctuation prediction | Hurt |
| Use the discrete-time Hessian approximation instead of the exact minibatch noise model | SGD fluctuation predictions · Linear regression with small batch size | fluctuation strength prediction | Hurt |
| Increase network width without reducing the learning rate | Two-layer tanh network trained with SGD · MNIST | training stability | Hurt |
| Increase learning rate while keeping batch size fixed | SGD power-law tail distribution · One-dimensional setting | tail index | Hurt |
Closest papers by meaning
- On the Noisy Gradient Descent that Generalizes as SGD (ICML 2020 · similarity 0.93)
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (ICML 2020 · similarity 0.92)
- Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent (ICML 2021 · similarity 0.92)
- The alignment property of SGD noise and how it helps select flat minima: A stability analysis (NeurIPS 2022 · similarity 0.92)
- Special Properties of Gradient Descent with Large Learning Rates (ICML 2023 · similarity 0.91)
- Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.