Research map / Optimization / Optimizers
Probabilistic Line Searches for Stochastic Optimization
Maren Mahsereci, Philipp Hennig
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| control SGD step size with the probabilistic line search instead of a fixed learning rate | SGD on 2-layer NN · MNIST and CIFAR-10 test error after 10 epochs | test error vs initial learning rate: initial learning rate across five orders of magnitude gives close to optimal error | Improved |
| add line search to SGD compared with best tuned fixed-rate SGD without decay | SGD · MNIST and CIFAR-10 | test error: best SGD instance just barely outperformed the line searches | No clear effect |
| pick learning rate from short exploratory experiment (10^3 steps) rather than line search with alpha0=1 | SGD · MNIST and CIFAR-10 | test error: alpha=0.75 (MNIST) and alpha=0.08 (CIFAR-10) best after 10^3 steps but not best after 10 epochs | Hurt |
| add line search to SGD with large step sizes | SGD · MNIST and CIFAR-10 training loss | training stability | Improved |
| add line search to SGD, effect on overfitting | SGD · CIFAR-10 and MNIST test vs train error | overfitting | No clear effect |
| Wolfe-condition based line search, line search length statistics | Probabilistic line search · MNIST | function evaluations per line search: ~1.4 function evaluations on average, 80%-90% terminate after first evaluation | Improved |
Closest papers by meaning
- Stopping Bayesian Optimization with Probabilistic Regret Bounds (NeurIPS 2024 · similarity 0.89)
- Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization (NeurIPS 2024 · similarity 0.87)
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.87)
- The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms (NeurIPS 2024 · similarity 0.87)
- Random Exploration in Bayesian Optimization: Order-Optimal Regret and Computational Efficiency (ICML 2024 · similarity 0.87)
- BayeSQP: Bayesian Optimization through Sequential Quadratic Programming (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
