Research map / Optimization / Implicit bias and generalization
Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that SGD batch-size noise gives no implicit-bias benefit in single-epoch online learning and that SGD trajectories track noiseless gradient descent ("golden path") in loss and function space.
Problem
Whether the implicit bias of SGD noise (small batch sizes), which helps generalization in multi-epoch offline training, also helps in single-epoch online learning where train and test distributions coincide.
Approach
Empirically contrasts online and offline regimes across batch sizes and shows that dropping or adding SGD noise mid-training makes the loss curve snap to or away from a translated low-noise trajectory; proposes the golden path hypothesis that SGD noisily follows the gradient descent path, with batch size affecting only computational cost. Supported by a quadratic-loss theorem showing expected SGD
Key results
- In online learning, lower SGD noise (larger batch) consistently reaches lower loss per gradient step on CIFAR-5m, ImageNet, and C4, while offline small-batch runs end at better loss due to implicit bias.
- After dropping SGD noise mid-training, the loss curve snaps within a short interval to a translated version of the low-noise run from initialization on ResNet-18/CIFAR-5m and GPT-2-small/C4.
- Average TV distance between a noise-dropped model and a low-noise model matches the distance between two independent low-noise seeds, while high-noise vs low-noise distance is larger.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase batch size to reduce SGD noise | ResNet-18 · CIFAR-5m online | loss | Helped |
| Increase batch size to reduce SGD noise | ConvNeXt-T · ImageNet online (10 epochs) | loss | Helped |
| Increase batch size to reduce SGD noise | GPT-2-small · C4 online | loss | Helped |
| Use higher SGD noise (smaller batch size) | ResNet-18 · CIFAR-5m offline | loss | Helped |
| Use higher SGD noise (smaller batch size) | ConvNeXt-T · ImageNet offline | loss | Helped |
| Use higher SGD noise (smaller batch size) | GPT-2-small · C4 offline | loss | Helped |
| Increase batch size from 32 to 512 during training (ablation) | ResNet-18 · CIFAR-5m online | loss trajectory alignment | Helped |
| Increase batch size from 64 to 512 during training (ablation) | GPT-2-small · C4 online | loss trajectory alignment | Helped |
| Decrease batch size to increase SGD noise during training (ablation) | ResNet-18 · CIFAR-5m online | loss | Hurt |
| Decrease batch size to increase SGD noise during training (ablation) | GPT-2-small · C4 online | loss | Hurt |
| Increase batch size from 32 to 512 (ablation) | ResNet-18 · CIFAR-5m online | average TV distance | Helped |
| Increase batch size from 64 to 512 (ablation) | GPT-2-small · C4 online | average TV distance | Helped |
Surprising findings
- Offline learning with small batch sizes eventually plateaus at higher loss than large-batch runs, in contrast to the usual belief that small batches find better minima.
- Increasing SGD noise mid-training causes an immediate loss increase that the model then recovers from, following a noisier trajectory.
Limitations
- Evaluated on only three datasets (CIFAR-5m, ImageNet, C4) and small models; generality is uncertain.
- Golden path hypothesis does not hold naively in weight space due to permutation symmetries and dead neurons.
- Theory covers only quadratic loss with additive Gaussian gradient noise.
Experimental setup
Datasets: CIFAR-5m ImageNet C4
Benchmarks: CIFAR-5m test ImageNet test C4 validation loss
Metrics: test loss early-stopping loss average total variation distance of softmax outputs
Baselines: offline learning with varying batch size independent low-noise seeds (TV baseline)
Builds on: Deep Bootstrap Edge of Stability
Closest papers by meaning
- Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate (ICLR 2021 · similarity 0.90)
- Strength of Minibatch Noise in SGD (ICLR 2022 · similarity 0.89)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.87)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.87)
- Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression (ICLR 2024 · similarity 0.87)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.