Research map / Optimization / Implicit bias and generalization
The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase width at fixed depth (ablation) | CNN-16 on CIFAR-10 · CIFAR-10 | training convergence speed | Helped |
| Increase depth at fixed width (ablation) | CNNs on CIFAR-10 · CIFAR-10 | training convergence speed | Hurt |
| Add skip connections (ablation) | CNNs with varying depth · CIFAR-10 | trainability | Helped |
| Add batch normalization (ablation) | CNNs with varying depth · CIFAR-10 | trainability | Helped |
| Add both skip connections and batch normalization (ablation) | CNNs with varying depth · CIFAR-10 | gradient confusion | Helped |
| Increase depth at fixed width (ablation) | MLPs on MNIST · MNIST | training loss | Hurt |
| Increase width at fixed depth (ablation) | MLPs on MNIST · MNIST | training loss | Helped |
| Increase depth at fixed width (ablation) | CNNs · CIFAR-100 | training loss | Hurt |
| Increase width at fixed depth (ablation) | CNNs · CIFAR-100 | minimum pairwise gradient cosine similarity | No clear effect |
| Increase width at fixed depth (ablation) | WRNs without batch normalization · CIFAR-10 and CIFAR-100 | training loss | Helped |
| Add both skip connections and batch normalization (ablation) | CNNs with varying depth · CIFAR-100 | trainability | Helped |
Closest papers by meaning
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.91)
- An Improved Analysis of Training Over-parameterized Deep Neural Networks (NeurIPS 2019 · similarity 0.90)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.86)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.