Research map / Optimization / Implicit bias and generalization
SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SGD rather than any regularized empirical risk minimizer | SCO with population-convex loss · SCO instance (single-pass vs RERM) | suboptimality: \Omega(1) lower bound for RERM; 1/\sqrt{n} rate for SGD | Helped |
| Use single-pass SGD instead of GD on training loss | SCO · SCO instance (GD separation) | suboptimality: 1/n^{0.5} for SGD; \Omega\prn[\Big]{\frac{1}{n^{5/12}}} lower bound for GD | Helped |
| Make k passes using adaptive step sizes and validation selection | SGD · SCO instance (multi-pass construction) | suboptimality: \Omega\prn[\Big]{\frac{1}{\sqrt{n}}} for single-pass SGD; O\prn[\Big]{\frac{1}{\sqrt{nk}}} for multi-pass SGD | Helped |
| Use multi-pass SGD rather than RERM | SCO multi-pass construction · SCO instance (multi-pass construction) | excess risk: O\prn[\Big]{\frac{1}{\sqrt{nk}}} for multi-pass SGD; \Omega(1) lower bound for RERM | Helped |
| Use SGD rather than RERM on convex distributions | Distribution-free learning problem · Distribution-free learning problem, convex distributions | excess risk: O\left(\frac{1}{\sqrt{n}}\right) for SGD; \Omega(c_n^2) lower bound for RERM | Helped |
| Use SGD rather than ERM | Two-layer diagonal ReLU network · Two-layer diagonal ReLU network | excess risk: O(1/\sqrt{n}) for SGD; ERM fails to find an O(1)-suboptimal point | Helped |
Closest papers by meaning
- Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions (NeurIPS 2022 · similarity 0.93)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.92)
- The Benefits of Implicit Regularization from SGD in Least Squares Problems (NeurIPS 2021 · similarity 0.91)
- Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (NeurIPS 2019 · similarity 0.91)
- Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate (ICLR 2021 · similarity 0.91)
- Rapid Overfitting of Multi-Pass SGD in Stochastic Convex Optimization (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.