Research map / Optimization / Implicit bias and generalization
The Heavy-Tail Phenomenon in SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase stepsize eta (equivalently ratio eta/b) | SGD on synthetic Gaussian linear regression · synthetic linear regression, d=100 | tail-index alpha | Hurt |
| increase batch-size b | SGD on synthetic Gaussian linear regression · synthetic linear regression | tail-index alpha | Helped |
| increase data variance sigma (curvature) | SGD on synthetic Gaussian linear regression · synthetic linear regression, d=100, eta=0.1, b=5 | tail-index alpha | Hurt |
| increase dimension d | SGD on synthetic Gaussian linear regression · synthetic linear regression, sigma=1.75 | tail-index alpha | Hurt |
| replace SGD with RMSProp | synthetic linear regression iterates · synthetic linear regression | tail-index alpha | Helped |
| use SVRG instead of SGD | synthetic linear regression iterates · synthetic linear regression | heavy-tailedness of stationary distribution | No clear effect |
| increase eta/b in SGD for fully connected 3-layer network | FCN on MNIST and CIFAR10 · MNIST/CIFAR10 FCN, per-layer tail-index | tail-index alpha | Hurt |
| increase eta/b in SGD for VGG (layers 2-8) | VGG on CIFAR10 · CIFAR10 VGG-11 layerwise | tail-index alpha | Hurt |
Closest papers by meaning
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.94)
- Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks (NeurIPS 2020 · similarity 0.93)
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks (ICML 2019 · similarity 0.93)
- Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent (NeurIPS 2023 · similarity 0.92)
- Error Analysis Affected by Heavy-Tailed Gradients for Non-Convex Pairwise Stochastic Gradient Descent (AAAI 2025 · similarity 0.90)
- Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.