Research map / Optimization / Implicit bias and generalization
The Benefits of Implicit Regularization from SGD in Least Squares Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unregularized tail-averaged SGD with a tuned constant stepsize instead of ridge regression | One-hot least squares · One-hot least squares | excess risk at the stated sample sizes: \Ns \ge \Nr | Helped |
| Use optimally tuned ridge regression instead of tuned SGD | A constructed one-hot least-squares instance · One-hot least squares | sample size required for comparable excess risk: \Nr \le \frac{\Ns^2}{\log^2(\Ns)} | Hurt |
| Use unregularized tail-averaged SGD with a tuned constant stepsize instead of ridge regression | Gaussian least squares · Gaussian least squares | excess risk at the stated sample sizes: N_{\mathrm{sgd}} \ge (1+R^2) \cdot \kappa(\Nr) \cdot \log( a) \cdot \Nr | Helped |
| Use unregularized tail-averaged SGD with a tuned constant stepsize instead of ridge regression | Gaussian least squares with rotation-invariant random ground truth · Gaussian least squares; random and rotation invariant \wb^* | excess risk at equal sample size: \Ns \ge \Nr | Helped |
| Use optimally tuned ridge regression instead of tuned SGD | A constructed Gaussian least-squares instance · Gaussian least squares | sample size required for comparable excess risk: \Nr \le \frac{\Ns^2}{\log^2(\Ns)} | Hurt |
| Use tuned SGD instead of ridge regression with the same sample size | Gaussian least squares with constant signal-to-noise ratio and small tail signal · Gaussian least squares | excess risk: \Ns\ge\Nr | Helped |
| Use tuned SGD rather than tuned ridge regression on the experiment's six Gaussian instances | Gaussian least-squares experiments · Gaussian least squares; six covariance and ground-truth combinations | samples required for the same population risk | Mixed |
Closest papers by meaning
- Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions (NeurIPS 2022 · similarity 0.93)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.92)
- Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression (NeurIPS 2024 · similarity 0.90)
- Risk Bounds of Accelerated SGD for Overparameterized Linear Regression (ICLR 2024 · similarity 0.90)
- Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression (ICML 2025 · similarity 0.89)
- Learning Curves of Stochastic Gradient Descent in Kernel Regression (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.