Research map / Optimization / Implicit bias and generalization
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Marginalize more of the implicit effect by increasing M | Networks trained with Gaussian noise injections · Training data | training performance | Helped |
| Replace implicit-effect gradient noise with Gaussian gradient noise | M=16 models · Training data | training performance | Helped |
| Train with the explicit regularizer instead of full Gaussian noise injections | Convolutional networks · Held-out data | test loss | Helped |
| Increase the number of multiplicative-GNI samples used to marginalize the implicit effect | Networks trained with multiplicative GNIs · Held-out data with mini-batches | held-out performance | Helped |
| Increase the variance of multiplicative injected noise | Networks trained with multiplicative GNIs · Implicit-effect gradients | tail heaviness and skewness | Mixed |
| Use nonlinear activations rather than linear activations | Networks with Gaussian noise injections · Gradient noise | skewness | Helped |
| Model implicit-effect gradient noise with asymmetric α-stable noise rather than Gaussian noise | SGD with Gaussian noise injections · Gradient-descent training paths | dynamics fit | Helped |
| Increase tail heaviness and asymmetry of the stable noise | Asymmetric stable SGD on a quartic potential · One-dimensional quartic-potential simulation | distance of stationary-distribution modes from local minima | Hurt |
Closest papers by meaning
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks (ICML 2019 · similarity 0.89)
- Explicit Regularisation in Gaussian Noise Injections (NeurIPS 2020 · similarity 0.89)
- Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD (ICML 2024 · similarity 0.87)
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.86)
- Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning (ICML 2024 · similarity 0.86)
- High-dimensional Limit of SGD for Diagonal Linear Networks (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.