Research map / Optimization / Implicit bias and generalization
Train faster, generalize better: Stability of stochastic gradient descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| halve the step size | SGD · CIFAR-10, MNIST, ImageNet, and Penn Treebank | generalization error | Helped |
| train for more steps | SGD · deep neural network experiments | parameter distance between runs differing by one example | Mixed |
| replace cross-entropy loss with squared loss (ablation) | LeNet-inspired model · MNIST | generalization error | Helped |
| replace cross-entropy loss with squared loss (ablation) | LeNet-inspired model · MNIST | parameter divergence | Helped |
| move the changed training example from late in training to the first step (ablation) | AlexNet · ImageNet | parameter divergence | Hurt |
| increase model size (ablation) | AlexNet · ImageNet | generalization error | No clear effect |
| add dropout to SGM | SGM · stability bounds | uniform stability | Helped |
| add ℓ₂ regularization or weight decay | SGM · non-convex optimization | uniform stability | Helped |
| clip or otherwise bound gradient norms | SGM · stability bounds | uniform stability | Helped |
Closest papers by meaning
- Stability and Generalization of Stochastic Gradient Methods for Minimax Problems (ICML 2021 · similarity 0.94)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.91)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.90)
- Error Analysis Affected by Heavy-Tailed Gradients for Non-Convex Pairwise Stochastic Gradient Descent (AAAI 2025 · similarity 0.90)
- Generalization Analysis of Biased Stochastic Gradient Methods for Minimax Problems (NeurIPS 2026 · similarity 0.90)
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.