Research map / Optimization / Optimizers
Parameter-free Clipped Gradient Descent Meets Polyak
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Inexact Polyak Stepsize, a parameter-free clipped/Polyak-style gradient descent that needs no stepsize or clipping tuning and whose convergence rate is asymptotically independent of L under (L0, L1)-smoothness.
Problem
Gradient descent and clipped gradient descent require tuning the stepsize and gradient clipping threshold; existing parameter-free methods only address the stepsize and rely on problem-specific quantities or decreasing stepsizes that lose independence from L under (L0, L1)-smoothness.
Approach
Shows Polyak stepsize under (L0, L1)-smoothness matches clipped GD's rate (asymptotically independent of L), then replaces f* with a known lower bound l* (zero for nonnegative losses) and scales the stepsize by 1/sqrt(T), returning the best iterate, so the method is parameter-free without decreasing stepsizes.
Key results
- Under (L0,L1)-smoothness, Polyak stepsize attains the same O(L0||x0-x*||^2/T + L L1^2||x0-x*||^4/T^2) rate as well-tuned clipped gradient descent.
- Inexact Polyak Stepsize achieves O((L0||x0-x*||^2+sigma^2)/sqrt(T) + ...) rate with rate asymptotically independent of L, while DecSPS and AdaSPS degrade as L1 grows on the synthetic function.
- On LSTM, Nano-GPT and T5, Inexact Polyak Stepsize consistently outperforms DecSPS and AdaSPS across architectures, averaged over three seeds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Polyak stepsize instead of gradient descent with the optimal fixed stepsize | Convex optimization under (L0, L1)-smoothness · (L0, L1)-smooth optimization | convergence rate | Helped |
| Use Inexact Polyak Stepsize with the lower bound l* and best-iterate output | Convex optimization under (L0, L1)-smoothness · (L0, L1)-smooth optimization | convergence rate dependence on L | Helped |
| Increase L1 | Gradient descent · Synthetic function | convergence | Hurt |
| Increase L1 | Polyak stepsize · Synthetic function | convergence | No clear effect |
| Increase L1 | DecSPS and AdaSPS · Synthetic function | convergence | Hurt |
| Increase L1 | Inexact Polyak Stepsize · Synthetic function | convergence | No clear effect |
| Use Inexact Polyak Stepsize instead of DecSPS or AdaSPS | Neural-network training · LSTM, Nano-GPT, and T5 | test loss | Helped |
| Use DoG instead of other parameter-free methods | Neural-network training · LSTM | test loss | Helped |
| Use DoG instead of other parameter-free methods | Neural-network training · Nano-GPT | test loss | Hurt |
| Use Polyak stepsize instead of other methods | Neural-network training · T5 | test loss | Helped |
| Use Polyak stepsize instead of other methods | Neural-network training · LSTM and Nano-GPT | training convergence | Hurt |
| Decrease the stepsize monotonically during training | Polyak-based parameter-free methods such as DecSPS and AdaSPS · (L0, L1)-smooth optimization | convergence rate dependence on L | Hurt |
Surprising findings
- Plain Polyak stepsize with f* set to zero diverges for LSTM and Nano-GPT, although it performs best among parameter-free methods on T5.
- DoG performs best among parameter-free baselines on LSTM but is very unstable on Nano-GPT.
Limitations
- Convergence rate in T is O(1/sqrt(T)), which is slower than the optimal rate and than clipped GD with tuned hyperparameters.
- Theory is for convex functions; neural network experiments do not satisfy the best-iterate requirement and only reach stationary points.
- Neural network experiments use small-scale models and a single synthetic function for validation.
Experimental setup
Datasets: Penn Treebank Shakespeare C4 synthetic quartic function
Benchmarks: synthetic quartic function Penn Treebank (LSTM) Shakespeare (Nano-GPT) C4 (T5)
Metrics: convergence of loss / function gap final test loss
Baselines: Gradient Descent Clipped SGD Polyak stepsize DecSPS AdaSPS DoG
Builds on: Polyak stepsize clipped gradient descent (L0,L1)-smoothness DecSPS AdaSPS
Closest papers by meaning
- Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity (ICLR 2025 · similarity 0.88)
- Improved Analysis of Clipping Algorithms for Non-convex Optimization (NeurIPS 2020 · similarity 0.88)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.87)
- Non-convex Stochastic Composite Optimization with Polyak Momentum (ICML 2024 · similarity 0.86)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.86)
- New Perspectives on the Polyak Stepsize: Surrogate Functions and Negative Results (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.