Research map / Optimization / Nonconvex and smooth optimization
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows AdaGrad/Adam-norm (with/without delay and momentum) can have inverse-power high-probability complexity under heavy-tailed noise, and proves that clipping both the momentum input and the adaptive stepsize scalar yields polylogarithmic-in-confidence high-probability bounds.
Problem
High-probability convergence of AdaGrad/Adam-type methods under heavy-tailed stochastic gradient noise (bounded alpha-th moment, alpha in (1,2]) is poorly understood; existing bounds have inverse-power dependence on the confidence level or require sub-Gaussian/bounded noise.
Approach
Shows a negative result for unclipped Adam/AdaGrad (with momentum and delayed variants) via a 1-D Huber-loss construction, then proposes clipping the stochastic gradient both in the momentum update and in the adaptive scalar b_t, deriving polylogarithmic high-probability bounds for convex and non-convex smooth problems.
Key results
- On a 1-D quadratic with heavy-tailed noise, clipped AdaGrad(D) variants show lower 10th-90th percentile error bands than unclipped counterparts over 100 seeds.
- On ALBERT Base v2 fine-tuning on CoLA, Clip-Adam shows a narrow 5th-95th percentile band while Adam shows poor high-probability convergence; on RTE, Clip-Adam performs similarly to Adam.
- Theory: clipped Adam/AdaGrad-norm achieve polylogarithmic dependence on 1/delta, with non-convex complexity matching Clip-SGD up to log factors; unclipped variants have inverse-power dependence on delta (T = Omega(poly(delta^{-1/2}, delta^{-1/3}))).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gradient clipping | AdaGrad · Quadratic problem with additive heavy-tailed noise | squared distance to solution | Helped |
| add layer-wise clipping | Adam · CoLa | validation loss | Helped |
| add layer-wise clipping | Adam · RTE | validation loss | No clear effect |
| add clipping | Adam-based methods · QNLI and CoLa | performance | Helped |
Surprising findings
- Even with bounded variance (alpha=2), unclipped Adam(D)/M-AdaGrad(D) can require inverse-power dependence on the confidence level, contrary to the expectation that adaptivity provides implicit clipping.
- Clipping must be applied inside the adaptive stepsize computation b_t, not only the momentum update, to prevent b_t from growing too fast.
Limitations
- Assumption requiring bounded moments on a set Q (and an extra boundedness assumption for methods without delay) is relatively restrictive
- Negative results established only for beta_2 = 1 - 1/T and specific 1-D constructions; the case alpha<2 for unclipped methods is only conjectured
- Experiments are limited to a synthetic quadratic and small fine-tuning tasks (ALBERT, RoBERTa Large)
Experimental setup
Datasets: CoLA RTE QNLI synthetic quadratic with heavy-tailed noise
Benchmarks: CoLA RTE QNLI synthetic quadratic
Metrics: squared distance to solution (quadratic) validation loss percentiles (fine-tuning) high-probability iteration complexity (theory)
Baselines: Adam AdaGrad Clip-Adam AdamD AdaGradD Clip-AdaGrad
Builds on: Clip-SGD AdaGrad Adam AdamD AdaGrad-Norm
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.89)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.89)
- Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients? (NeurIPS 2026 · similarity 0.89)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.88)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.