Research map / Optimization / Optimizers
Generalized Polyak Step Size for First Order Optimization with Momentum
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adaptive Polyak-style step size for SGD with momentum (ALR-SMAG) instead of constant step size | SGDM on CIFAR100 WRN-28-10 · CIFAR100 test accuracy | test accuracy (%): 75.36 to 76.51 | Helped |
| adaptive step size ALR-SMAG vs step-decay on DenseNet121 | SGDM on CIFAR100 DenseNet121 · CIFAR100 test accuracy | test accuracy (%): 75.12 to 75.25 | Helped |
| adaptive step size ALR-SMAG vs step-decay on WRN-28-10 without warmup | SGDM on CIFAR100 WRN-28-10 · CIFAR100 test accuracy | test accuracy (%): 76.51 vs 76.49 | No clear effect |
| warmup on the upper bound of the step size (ablation) | ALR-SMAG and ALR-SHB · CIFAR100 WRN-28-10 and DenseNet121 | test accuracy (%): ALR-SMAG 76.51 to 77.63 on WRN; ALR-SHB 76.36 to 77.57 | Helped |
| L4Mom step size instead of the proposed adaptive step size | SGD with momentum on CIFAR100 WRN-28-10 · CIFAR100 test accuracy | test accuracy (%): 66.84 vs 76.51 for ALR-SMAG | Hurt |
| decoupled weight decay added to ALR-SMAG search direction (ablation) | ALR-SMAG on CIFAR100 WRN-28-10 · CIFAR100 with weight decay test accuracy | test accuracy (%): best 81.18 with WD, below SGDM-STEP 81.22 and SGDM-COSINE 81.85 at best | Helped |
| final fine-tuning phase that increases c exponentially in last 20% of iterations (ablation) | ALR-SMAG with weight decay · CIFAR100 WRN-28-10 with WD | test accuracy (%): 81.18 to 81.89 | Helped |
| ALR-HB (v2) step size with 1/(2L) constant term vs heavy ball with tuned parameters on least squares (kappa=1e4) | heavy-ball on least squares · least-squares synthetic | final loss sensitivity to beta | Helped |
| adaptive step size ALR-SMAG vs SGDM with step-decay on CIFAR100 WRN-28-10 with weight decay (ALI-G with momentum comparison) | ALR-SMAG with weight decay · CIFAR100 WRN-28-10 with WD | test accuracy (%): 81.18 vs 81.22 (SGDM-STEP) and 81.85 (SGDM-COSINE) | No clear effect |
Closest papers by meaning
- Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance (ICLR 2025 · similarity 0.91)
- Non-convex Stochastic Composite Optimization with Polyak Momentum (ICML 2024 · similarity 0.89)
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.87)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.87)
- Parameter-free Clipped Gradient Descent Meets Polyak (NeurIPS 2024 · similarity 0.86)
- New Perspectives on the Polyak Stepsize: Surrogate Functions and Negative Results (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.