Research map / Optimization / Optimizers
Multiplicative Noise and Heavy Tails in Stochastic Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Introduce multiplicative noise alongside additive noise | Perturbed GD · Non-convex 1D objective | basin hopping | Helped |
| Use SGD with single-sample batches rather than perturbed GD with additive noise | Two-layer neural network, 16 hidden units · Musk | basin hopping | Helped |
| Increase step size from 0.001 to 0.025 (ablation) | SGD · Wine Quality | estimated tail exponent: 4.12 $\pm$ 0.04 -> 2.97 $\pm$ 0.03 | Hurt |
| Decrease minibatch size from 10 to 1 (ablation) | SGD · Wine Quality | estimated tail exponent: 5.99 $\pm$ 0.05 -> 2.97 $\pm$ 0.03 | Hurt |
| Increase L^2 regularization from 10^{-4} to 0.2 (ablation) | SGD · Wine Quality | estimated tail exponent: 2.97 $\pm$ 0.03 -> 2.55 $\pm$ 0.01 | No clear effect |
| Add Gaussian input noise with standard deviation 1 (ablation) | SGD · Wine Quality | estimated tail exponent: 2.97 $\pm$ 0.03 -> 2.36 $\pm$ 0.13 | Helped |
| Add momentum with \eta = 0.5 (ablation) | SGD · Wine Quality | estimated tail exponent: 2.93 $\pm$ 0.03 -> 4.99 $\pm$ 0.03 | Mixed |
| Replace SGD with Adam (ablation) | Two-layer neural network, four hidden units · Wine Quality | estimated tail exponent: 2.93 $\pm$ 0.03 -> 2.119 $\pm$ 0.005 | Mixed |
| Replace SGD with SSN (ablation) | Two-layer neural network, four hidden units · Wine Quality | estimated tail exponent: 2.93 $\pm$ 0.03 -> 0.79 $\pm$ 0.04 | Helped |
| Use multiplicative noise rather than additive light-tailed noise | SGD · CIFAR10 | heavy-tailed step norms: \hat{\alpha} = 1.52 \pm 0.01; p < 10^{-8} | Helped |
| Increase step size | Perturbed GD with multiplicative noise · Non-convex 1D objective | tail heaviness | Helped |
Closest papers by meaning
- Stochastic Zeroth-Order Optimization Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.91)
- Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping (ICLR 2025 · similarity 0.90)
- Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits (NeurIPS 2025 · similarity 0.89)
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.89)
- Generalization Bounds for Heavy-Tailed SDEs through the Fractional Fokker-Planck Equation (ICML 2024 · similarity 0.88)
- Error Analysis Affected by Heavy-Tailed Gradients for Non-Convex Pairwise Stochastic Gradient Descent (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.