Research map / Optimization / Implicit bias and generalization
Step Size Matters in Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the gradient descent step size from 2·10^{-4} to 3·10^{-4} | Two-layer ReLU network · Piecewise continuous function estimation | convergence behavior: 2\cdot 10^{-4} -> 3\cdot 10^{-4} | Hurt |
| Use identity initialization for all weight matrices | Deep linear network · Symmetric positive semidefinite matrix estimation | convergence to the global optimum: \delta \le \min\left\{ \frac{1}{L}, \ \frac{1}{L\rho(R)^{2(L-1)/L}} \right\} | Helped |
| Use identity initialization for all weight matrices | Deep linear network · Symmetric matrix estimation with a negative eigenvalue | distance to the closest positive semidefinite matrix: \delta \le \min\left\{ \frac{1}{1 - \lambda_{\min}(R)}, \frac{1}{L}, \frac{1}{L\rho(R)^{2(L-1)/L}} \right\} | Helped |
| Use a larger-singular-value target matrix | Deep linear network trained with gradient descent · Linear mapping estimation | maximum step size for convergence to a global optimum: \delta \le \frac{2}{L\rho(R)^{2(L-1)/L}} | Hurt |
| Train with a large step size | Deep linear network trained with gradient descent · Linear mapping estimation | largest singular value of the estimated mapping: \rho(\hat R) \le \left(\frac{2}{L\delta}\right)^{L/(2L-2)} | Helped |
| Train with a large step size | Two-layer ReLU network · Training examples | maximum product of input and estimated-output norms: \max_{i \in [N]} \|x_i\|_2\|\hat f(x_i)\|_2 \le \frac{1}{\delta} | Helped |
Closest papers by meaning
- Flatland: The Adventures of Gradient Descent with Large Step Sizes (ICML 2026 · similarity 0.91)
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.88)
- Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization (NeurIPS 2024 · similarity 0.87)
- Understanding Optimization in Deep Learning with Central Flows (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.