Research map / Optimization / Implicit bias and generalization
Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the learning-rate-to-batch-size ratio (η/b) | SGD-trained FCN4, FCN6, and CNN models · MNIST and CIFAR10 | estimated tail index (α̂) | Helped |
| Use heavier-tailed network weights | Layer-wise magnitude pruning and singular value pruning · MNIST and CIFAR10 | relative test accuracy after pruning | Helped |
| Use heavier-tailed network weights | Trained neural networks · MNIST and CIFAR10 | test accuracy | Helped |
| Increase network width under the heavy-tailed mean-field limit assumptions | Fully connected networks · Theoretical compression guarantees | relative compression error | Helped |
| Prune by keeping the largest-magnitude weights | Heavy-tailed mean-field networks · Global and layer-wise magnitude pruning | relative ℓp compression error | Helped |
| Prune by keeping the largest singular values | Heavy-tailed mean-field networks with stable limiting weights · Singular value pruning | relative weight error | Helped |
| Prune columns with the smallest ℓp norms | Heavy-tailed mean-field networks · Node pruning | relative ℓp compression error | Helped |
| Increase the pruning fraction while holding relative compression error fixed | Compressible networks · Generalization bound | generalization error bound | Helped |
| Use correlated rather than independent layer parameters (ablation) | Synthetic heavy-tailed weight matrices · Synthetic pruning experiment | prunability advantage from heavier tails | Hurt |
| Apply node pruning to FCNs | FCN models · MNIST and CIFAR10 | relative test accuracy after pruning | Mixed |
| Use the alternative univariate tail-index estimator (ablation) | Tail-index estimation · MNIST and CIFAR10 experiments | qualitative experimental results | No clear effect |
| Center pruned parameters with the mean instead of the median (ablation) | Magnitude, singular value, and node pruning experiments · MNIST and CIFAR10 | qualitative pruning results | No clear effect |
Closest papers by meaning
- Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD (ICML 2024 · similarity 0.95)
- The Heavy-Tail Phenomenon in SGD (ICML 2021 · similarity 0.91)
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.90)
- Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent (NeurIPS 2023 · similarity 0.89)
- Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate (ICLR 2024 · similarity 0.87)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.