Research map / Optimization / Implicit bias and generalization
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase fully connected network size | SGD on MNIST · MNIST | tail-index α | Mixed |
| Increase fully connected network size | SGD on CIFAR10 · CIFAR10 | tail-index α: 1.1–1.2 | No clear effect |
| Increase AlexNet width from an under-parametrized regime | SGD on CIFAR10 · CIFAR10 | tail-index α: 1.0–1.1 | Mixed |
| Increase AlexNet width from an under-parametrized regime | SGD on CIFAR100 · CIFAR100 | tail-index α: 1.0–1.1 | Mixed |
| Increase minibatch size | SGD with a fully connected network · MNIST and CIFAR10 | tail-index α: almost the same; a small interval | No clear effect |
| Switch between negative-log-likelihood and linear hinge loss | SGD across the evaluated network and dataset configurations · MNIST, CIFAR10, and CIFAR100 | tail-index α | No clear effect |
| Change random initialization seed | SGD across the evaluated network and dataset configurations · MNIST, CIFAR10, and CIFAR100 | tail-index α | No clear effect |
| Use the observed heavy-tailed gradient noise instead of assuming Gaussian noise | SGD noise modeling across the evaluated configurations · MNIST, CIFAR10, and CIFAR100 | tail-index α | Helped |
| Train through the early phase where α decreases to its lowest value | SGD with a fully connected network · MNIST and CIFAR10 | accuracy | Mixed |
Closest papers by meaning
- First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise (NeurIPS 2019 · similarity 0.93)
- The Heavy-Tail Phenomenon in SGD (ICML 2021 · similarity 0.93)
- On the Overlooked Structure of Stochastic Gradients (NeurIPS 2023 · similarity 0.92)
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.90)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.89)
- High-dimensional Limit of SGD for Diagonal Linear Networks (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.