Research map / Optimization / Implicit bias and generalization
Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Disentangles SGD's implicit bias from architectural bias by comparing SGD-trained and randomly sampled (Guess and Check) zero-training-error LeNet/MLP/ResNet networks on binary MNIST/CIFAR10 low-sample tasks, varying width and depth.
Problem
Whether good generalization of overparameterized neural networks comes from SGD's implicit bias or from the architecture's inherent bias toward simple functions, as claimed by the volume hypothesis / Guess and Check study.
Approach
Shows that the prior conclusion of similar SGD and random-sampling performance stems from suboptimal uniform [-1,1] initialization of SGD; with Kaiming initialization, increasing width helps SGD but not random sampling (SGD bias), while increasing depth hurts both (architectural bias). Proposes a Lipschitz-normalized loss in place of the weight-Frobenius normalization, which is not comparable acro
Key results
- SGD with Kaiming uniform init reaches 96.3% test accuracy on 16-sample MNIST 0 vs 7, versus 83.6% for SGD with Uniform[-1,1] and 82.8% for Guess and Check.
- Increasing width improves SGD test accuracy while Guess and Check stays roughly constant, indicating a positive optimization bias of SGD.
- Increasing depth reduces test accuracy for both SGD and Guess and Check, which is attributed to a negative architectural bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Kaiming initialization instead of Uniform [-1,1] | SGD on LeNet · MNIST 0 vs 7, 16 training samples | test accuracy: 83.6% vs. 82.8%; 96.3% | Helped |
| Increase network width (ablation) | SGD · MNIST and CIFAR10, low-sample binary classification | test accuracy | Helped |
| Increase network width (ablation) | Guess and Check · MNIST and CIFAR10, low-sample binary classification | test accuracy | No clear effect |
| Increase network depth (ablation) | SGD · MNIST and CIFAR10, low-sample binary classification | test accuracy | Mixed |
| Increase network depth (ablation) | Guess and Check · MNIST and CIFAR10, low-sample binary classification | test accuracy | Hurt |
| Initialize SGD with a perfectly fitting network sampled by Guess and Check (ablation) | SGD · MNIST and CIFAR10 binary classification | test accuracy | Helped |
| Use Uniform [-1,1] initialization instead of proper Kaiming initialization (ablation) | SGD · MNIST 0 vs 7, 16 training samples | test accuracy: 83.6% vs. 82.8%; 96.3% | Hurt |
Surprising findings
- The previously reported similarity between SGD and random sampling is largely an artifact of suboptimal SGD initialization.
- The weight-normalized loss used by prior work gives conclusions that reverse under Lipschitz normalization when comparing widths.
Limitations
- Restricted to binary classification in the low-sample regime because random sampling is exponentially costly.
- Lipschitz constant is lower-bounded using data points only, so the normalized loss is an estimate.
- Does not claim depth is always harmful; with more data deeper networks may generalize well, which is not tested.
Experimental setup
Datasets: MNIST CIFAR10
Benchmarks: MNIST binary class pairs CIFAR10 binary class pairs
Metrics: test accuracy Lipschitz normalized loss weight normalized loss negative log probability of zero training error
Baselines: Guess and Check (random sampling) SGD with Uniform[-1,1] initialization
Builds on: volume hypothesis Guess and Check Kaiming initialization
Closest papers by meaning
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.92)
- How much does Initialization Affect Generalization? (ICML 2023 · similarity 0.91)
- The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent (ICML 2020 · similarity 0.91)
- How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers (ICML 2024 · similarity 0.89)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.89)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.