Research map / Optimization / Implicit bias and generalization
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase network width | MLPs, CNNs, and ResNets without batch normalization · MNIST, Fashion-MNIST, and CIFAR-10 | test accuracy | Helped |
| increase width and tune SGD to its optimal normalized noise scale | networks without batch normalization · MNIST, Fashion-MNIST, and CIFAR-10 | optimal normalized noise scale | Helped |
| increase width | NTK-parameterized WRNs with batch normalization · CIFAR-10 | test accuracy | Helped |
| increase width | NTK-parameterized CNNs with batch normalization · Fashion-MNIST and CIFAR-10 | test accuracy | Helped |
| increase width | CNNs with batch normalization · batch-normalized CNN experiments | optimal learning rate or batch size trend | Mixed |
| train at a fixed noise scale below the optimum rather than increasing it with width | networks without batch normalization · test accuracy across network widths | test accuracy | No clear effect |
| add dropout, label smoothing, and data augmentation (ablation) | standard-parameterized WRNs · CIFAR-10 | test accuracy | Helped |
| add dropout, label smoothing, and data augmentation (ablation) | standard-parameterized WRNs · CIFAR-10 | optimal normalized noise scale versus width | No clear effect |
| use standard rather than NTK parameterization | WRNs and CNNs · CIFAR-10 WRNs and Fashion-MNIST CNNs | peak test accuracy | Helped |
| use standard rather than NTK parameterization | MLPs with equal-width hidden layers · MNIST | peak test accuracy | No clear effect |
| use standard rather than NTK parameterization | bottom-heavy and top-heavy 3-layer perceptrons · MNIST | peak test accuracy | Mixed |
Closest papers by meaning
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (ICML 2020 · similarity 0.90)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.87)
- Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods (ICLR 2025 · similarity 0.87)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.