Research map / Optimization / Nonconvex and smooth optimization
Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use deterministic gradient clipping with a fixed positive threshold | Gradient descent · Non-convex deterministic optimization | average gradient norm: The bound is \cO\left(\sqrt{\frac{F_0}{\eta T}} + \frac{F_0}{\eta T c}\right) | Mixed |
| Use deterministic gradient clipping | Gradient descent · Strongly convex deterministic optimization | iteration complexity: The bound includes \frac{R_0 }{c \eta} \min\left(\sqrt{\frac{L}{\mu}}, \frac{L R_0}{c}\right) | Hurt |
| Use stochastic gradient clipping under bounded variance rather than uniform boundedness | SGD · Non-convex stochastic optimization | minimum expected gradient norm: \min\Big\{\sigma,\frac{ \sigma^2}{c}\Big\} | Hurt |
| Increase the clipping threshold c | Clipped SGD · Non-convex stochastic optimization | bias-induced gradient norm neighborhood: \min\{\sigma, \sigma^2 / c\} | Helped |
| Decrease the clipping threshold below the target gradient norm | Deterministic clipped gradient descent · w1a | convergence speed | Hurt |
| Use smaller clipping radii | Deterministic clipped gradient descent · w1a | best tuned step-size | Helped |
| Reduce the step-size while keeping clipping | Clipped SGD · Quadratic function with χ²(1) noise and w1a | gradient norm | No clear effect |
| Use clipping instead of vanilla SGD | SGD · Quadratic function with χ²(1) noise and w1a | gradient norm | Hurt |
Closest papers by meaning
- Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Stochastic Bias and Remedies (ICML 2025 · similarity 0.92)
- Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions (NeurIPS 2025 · similarity 0.91)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.91)
- To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.91)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.90)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.