Research map / Optimization / Implicit bias and generalization
On the Origin of Implicit Regularization in Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the minibatch-gradient-norm implicit regularizer to the loss | 10-1 Wide-ResNet trained with small learning rates · CIFAR-10 | test accuracy | Helped |
| Set the explicit regularization coefficient to 2^{-6} | Modified-loss training · CIFAR-10 | test accuracy | Helped |
| Increase the learning rate on the original loss from the training-accuracy optimum to the test-accuracy optimum | 10-1 Wide-ResNet · CIFAR-10 | test accuracy: 2^{-10} -> 2^{-6} | Helped |
| Add the explicit implicit regularizer with λ = 2^{-6} | 10-1 Wide-ResNet · CIFAR-10 | test accuracy versus training accuracy: 2^{-8} and 2^{-8} | No clear effect |
| Scale the learning rate in proportion to batch size | 10-1 Wide-ResNet trained on the original loss · CIFAR-10 | test accuracy | No clear effect |
| Scale the explicit regularization coefficient in proportion to batch size | 10-1 Wide-ResNet trained on the modified loss · CIFAR-10 | test accuracy | No clear effect |
| Increase n from 1 to 16 sequential updates per minibatch (ablation) | SGD · CIFAR-10 | test accuracy: 93.5% -> 88.8% | Hurt |
| Increase n while holding the bare learning rate α constant (ablation) | n-step SGD · Fashion-MNIST | test accuracy | No clear effect |
| Increase the explicit regularization coefficient beyond its optimum (ablation) | Modified-loss training · CIFAR-10 | test accuracy | Hurt |
| Increase the learning rate beyond its optimum (ablation) | SGD on the original loss · CIFAR-10 | test accuracy | Hurt |
Closest papers by meaning
- Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions (NeurIPS 2022 · similarity 0.92)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.92)
- Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks (ICLR 2018 · similarity 0.91)
- Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent (ICLR 2023 · similarity 0.91)
- Tighter Convergence Bounds for Shuffled SGD via Primal-Dual Perspective (NeurIPS 2024 · similarity 0.88)
- Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.