Research map / Optimization / Optimizers
Barzilai-Borwein Step Size for Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace hand-tuned fixed step size with Barzilai-Borwein step size computed each epoch | SVRG (SVRG-BB) · rcv1.binary, w8a, ijcnn1 (LR/SVM) | sub-optimality F(x)-F(x*) vs epochs | No clear effect |
| Replace best-tuned SGD step size with BB step size plus smoothing (SGD-BB) | SGD · rcv1.binary, w8a, ijcnn1 | sub-optimality vs epochs | No clear effect |
| Use SGD-BB instead of SGD with fixed diminishing step sizes eta/(k+1) that are not best-tuned | SGD · rcv1.binary, w8a, ijcnn1 | sub-optimality vs epochs | Helped |
| Use SVRG-BB with BB step size instead of fixed step sizes not best-tuned | SVRG · rcv1.binary, w8a, ijcnn1 | sub-optimality vs epochs | Helped |
| Add BB step size to SGD-BB vs AdaGrad comparison | SGD-BB vs AdaGrad · rcv1.binary, w8a, ijcnn1 | CPU time to reach sub-optimality | Helped |
| Use SAG-BB (BB step size with smoothing) instead of SAG with line search | SAG · rcv1.binary, w8a, ijcnn1 | CPU time to reach sub-optimality | Helped |
| Use SGD-BB instead of oLBFGS (best-tuned step size) | stochastic quasi-Newton oLBFGS · rcv1.binary, w8a, ijcnn1 | CPU time to reach sub-optimality | Helped |
| Apply smoothing technique to BB step sizes in SGD-BB | SGD-BB · rcv1.binary, w8a, ijcnn1 | stability of step sizes | Helped |
Closest papers by meaning
- Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (NeurIPS 2019 · similarity 0.89)
- Revisit last-iterate convergence of mSGD under milder requirement on step size (NeurIPS 2022 · similarity 0.88)
- An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Bias (NeurIPS 2021 · similarity 0.88)
- Derivatives of Stochastic Gradient Descent in parametric optimization (NeurIPS 2024 · similarity 0.87)
- Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance (ICLR 2025 · similarity 0.87)
- High-dimensional Limit of SGD for Diagonal Linear Networks (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.