Research map / Optimization / Implicit bias and generalization
Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase SGD temperature in the lazy regime | FC and CNN classifiers · MNIST and CIFAR10 test sets | test error | Mixed |
| Increase training-set size P in the lazy regime | FC and CNN classifiers · MNIST and CIFAR10 | test error | Mixed |
| Increase SGD temperature T | FC and CNN classifiers · MNIST and CIFAR10, lazy and feature regimes | relative weight variation: Δw ~ T^delta P^gamma | Helped |
| Increase SGD temperature T | FC and CNN classifiers · MNIST and CIFAR10 | training time: t* ~ T P^b | Hurt |
| Increase training-set size P | FC and CNN classifiers · MNIST and CIFAR10 | training time: t* ~ T P^b | Hurt |
| Increase training-set size P | SGD-trained neural networks · MNIST and CIFAR10 | characteristic temperature T_c: T_c ~ P^-a | Hurt |
| Decrease initialization scale alpha instead of increasing SGD temperature | FC and CNN classifiers · MNIST and CIFAR10, lazy regime | test error | Mixed |
| Replace hinge loss trained to zero loss with cross-entropy and early stopping | FC on MNIST · MNIST | power-law dependencies on T and P | No clear effect |
| Increase data-distribution parameter chi | Perceptron · Perceptron classification | gamma exponent for weight variation versus P: gamma = 1/(1+chi) | Hurt |
Closest papers by meaning
- Strength of Minibatch Noise in SGD (ICLR 2022 · similarity 0.91)
- Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks (NeurIPS 2023 · similarity 0.89)
- Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning (ICML 2024 · similarity 0.89)
- Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate (ICLR 2024 · similarity 0.88)
- Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression (ICLR 2024 · similarity 0.88)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.