Research map / Optimization / Implicit bias and generalization
SGD with Large Step Sizes Learns Sparse Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Extend the large-step-size loss-stabilization phase before decaying the step size | SGD training schedules · Diagonal linear networks, ReLU networks, and deep networks | generalization | Helped |
| Use a large-step-size schedule instead of a small-step-size schedule | DenseNet-100-12 · CIFAR-100, basic setting | test error: 60% -> 35% | Helped |
| Use a large-step-size schedule with loss stabilization | ResNet-18 · CIFAR-10 | test error: 35% -> 12% | Helped |
| Extend the large-step-size loss-stabilization phase | SGD-trained neural networks · Diagonal linear, ReLU, and deep-network experiments | feature sparsity | Helped |
| Remove the noise term from the SDE discretization (ablation) | SDE model of SGD · Diagonal linear and simple ReLU experiments | feature sparsity | Hurt |
| Use a larger step size, even when it prevents fitting all training points | Two-layer ReLU network · 1D regression | training fit | Hurt |
Closest papers by meaning
- Special Properties of Gradient Descent with Large Learning Rates (ICML 2023 · similarity 0.93)
- Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks (NeurIPS 2023 · similarity 0.92)
- Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks (ICLR 2018 · similarity 0.91)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.91)
- On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (ICLR 2019 · similarity 0.91)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.