Research map / Optimization / Implicit bias and generalization
To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes clipped streaming SGD on high-dimensional least squares via a deterministic ODE limit (clipped homogenized SGD), derives stability and speedup criteria based on clipping reduction factors, and proposes a heuristic clipping schedule.
Problem
Understanding when and how gradient clipping helps SGD in high dimensions, and how the clipping threshold should scale with problem size, under streaming linear regression with heavy-tailed or Gaussian noise.
Approach
Shows that in high intrinsic dimension, clipped SGD risk curves are described by an SDE (clipped homogenized SGD) with descent and variance reduction factors mu_c and nu_c, yielding a deterministic ODE; from this derives a clipping-comparison criterion and a one-hyperparameter heuristic clipping schedule c proportional to sqrt(2R+sigma^2) with compensated learning rate.
Key results
- Deterministic ODE closely matches clipped SGD risk curves on synthetic (Gaussian and Cauchy noise), CIFAR10 and Wikitext2 linear models.
- With Gaussian noise clipping provably never improves over unclipped SGD (mu^2/nu <= 1 for all R,c).
- Heuristic schedule c=kappa*sqrt(2R+sigma^2) with compensated learning rate nearly matches the optimal max-criterion schedule under Rademacher-like noise, with only one tuned scalar kappa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| clip gradients with Gaussian noise | streaming SGD · Gaussian data and Gaussian noise | risk | No clear effect |
| clip gradients with Rademacher-like noise | streaming SGD · Rademacher-like noise | risk | Helped |
| clip gradients with heavy-tailed noise | streaming SGD · Cauchy noise | convergence | Helped |
| use the heuristic clipping and compensated learning-rate schedules instead of the max-criterion schedules | clipped SGD · Gaussian and Rademacher-like noise | risk | No clear effect |
| clip gradients with a threshold satisfying the clipped-stability criterion | streaming SGD · Gaussian data | stability | Helped |
| clip gradients in the stated discrete-noise regime where the reduction-factor ratio is below 1 | streaming SGD · Gaussian data with discrete three-point noise | stability | Hurt |
Surprising findings
- Clipping can decrease stability relative to unclipped SGD under some noise distributions.
- Under Gaussian noise, clipping cannot improve SGD performance at any threshold.
Limitations
- Analysis restricted to linear regression with streaming SGD and a high-dimensional limit.
- Non-linear extension (ResNet18, ViT) only demonstrated as a technique, not fully validated theoretically.
- Heuristic schedule validated only in a synthetic setting; thorough exploration left to future work.
Experimental setup
Datasets: CIFAR10 Wikitext2 synthetic Gaussian data
Benchmarks: CIFAR10 Wikitext2 synthetic least squares
Metrics: risk distance to optimality
Baselines: unclipped SGD max-clipping-criterion optimal schedule
Builds on: homogenized SGD clipped SGD analysis under heavy-tailed noise
Closest papers by meaning
- Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Stochastic Bias and Remedies (ICML 2025 · similarity 0.90)
- Near-Optimal Streaming Heavy-Tailed Statistical Estimation with Clipped SGD (NeurIPS 2024 · similarity 0.88)
- Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions (NeurIPS 2025 · similarity 0.88)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.87)
- Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients? (NeurIPS 2026 · similarity 0.87)
- Dimension-adapted Momentum Outscales SGD (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.