Research map / Optimization / Implicit bias and generalization
Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stochastic gradient flow instead of gradient flow with the same initialization | Diagonal linear networks · Sparse regression | validation loss | Helped |
| Increase the integral of training loss along the stochastic trajectory | Stochastic gradient flow · Sparse regression | validation loss | Helped |
| Inject label noise during SGD training | Diagonal linear networks · Sparse regression | sparsity of the selected solution | Helped |
| Add too much label noise | Diagonal linear networks trained with SGD · Sparse regression | convergence | Hurt |
| Use larger initialization scale | Gradient flow on diagonal linear networks · Overparametrized noiseless regression | sparsity of the selected solution | Hurt |
| Use larger initialization scale | Gradient flow on diagonal linear networks · Overparametrized noiseless regression | training time | Helped |
| Use the stochastic gradient flow model in place of discrete SGD | Diagonal linear networks · Sparse regression | validation loss trajectory similarity | Helped |
| Use isotropic noise instead of noise structured like intrinsic SGD noise | Stochastic gradient flow · Overparametrized noiseless regression | implicit-bias characterization | Hurt |
Closest papers by meaning
- (S)GD over Diagonal Linear Networks: Implicit bias, Large Stepsizes and Edge of Stability (NeurIPS 2023 · similarity 0.94)
- High-dimensional Limit of SGD for Diagonal Linear Networks (NeurIPS 2026 · similarity 0.92)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.90)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.90)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.90)
- Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.