Research map / Optimization / Optimizers
Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models
Leonardo Galli, Holger Rauhut, Mark Schmidt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a nonmonotone rather than monotone stochastic line search | SGD/Adam with line search · Over-parameterized model experiments | convergence speed | Improved |
| Use PoNoS instead of SLS | SGD with line search · Image classification experiments | test accuracy | Improved |
| Use PoNoS instead of SGD or Adam | Image classification experiments · Image classification experiments | train loss | Improved |
| Use PoNoS instead of SGD | Image classification training · Image classification experiments | total computational time | Improved |
| Combine the nonmonotone line search with the Polyak initial step size | PoNoS · Mushrooms | iterations to reach train loss 10^{-4}: 6 times the iterations | Improved |
| Use PoNoS_prec instead of other tested methods | Preconditioned transformer training · PTB and Wikitext2 | train loss convergence speed | Mixed |
| Use the new resetting technique (ablation) | PoNoS · Image classification experiments | number of backtracks: 0 | Improved |
| Remove the new resetting technique (ablation) | PoNoS_reset0 · Image classification experiments | train loss | No clear effect |
| Compute a new step only every 2 iterations (ablation) | PoNoS_reset0 · Image classification experiments | train loss | Hurt |
| Use the alternative Zhang resetting initial step sizes (ablation) | Nonmonotone line search · Image classification experiments | train loss | Hurt |
| Use the smoothed Polyak initial step size instead of the non-smoothed version (ablation) | SPS · MLP and FashionMNIST | convergence speed | Hurt |
| Set the line-search parameter c to 0.1 rather than a larger value (ablation) | Stochastic line search · Experiments | stability: 0.1 | Hurt |
Closest papers by meaning
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.89)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.88)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.88)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.87)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.87)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
