Research map / Optimization / Optimizers
Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking
Frederik Kunstner, Victor S. Portella, Mark Schmidt, Nick Harvey
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multidimensional backtracking instead of diagonal AdaGrad, diagonal BB, RPROP, GD-HD with line-search | Gradient descent with line-search · cpusmall and breast-cancer (ill-conditioned) | convergence per function/gradient evaluation | Improved |
| Use multidimensional backtracking instead of plain gradient descent | Gradient descent · News20 (d ~ 10^6) | convergence per oracle call | Improved |
| Use diagonal Hessian preconditioning instead of no preconditioning | Gradient descent · cpusmall | condition number: 5\cdot 10^{13} -> 300 | Improved |
| Use ellipsoid variant of multidimensional backtracking instead of the globally optimal preconditioner with line-search | Gradient descent on ill-conditioned linear regression · Synthetic linear regression | convergence | Improved |
| Use plain gradient updates on the step-size hyperparameters (GD-HD) instead of a robust method | Hypergradient descent · Ill-conditioned problems | stability | Hurt |
| Use diagonal BB or RPROP instead of diagonal Hessian preconditioning | Gradient descent · Some ill-conditioned problems | convergence | Mixed |
| Use multidimensional backtracking instead of gradient descent on a less ill-conditioned problem | Gradient descent · News20 | relative advantage | Mixed |
| Lower GD-HD initial step-size from 10^-3 to 10^-10 | GD-HD · All experiments | divergence: 10^{-3} -> 10^{-10} | Improved |
| Add line-search to diagonal AdaGrad | Diagonal AdaGrad · Deterministic setting | competitiveness | Improved |
Closest papers by meaning
- Adaptive backtracking line search (ICLR 2025 · similarity 0.90)
- Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis (NeurIPS 2025 · similarity 0.86)
- Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization (NeurIPS 2024 · similarity 0.86)
- Revisiting Frank-Wolfe for Structured Nonconvex Optimization (NeurIPS 2025 · similarity 0.85)
- Escaping saddle points without Lipschitz smoothness: the power of nonlinear preconditioning (NeurIPS 2025 · similarity 0.85)
- Structured Preconditioners in Adaptive Optimization: A Unified Analysis (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
