Research map / Optimization / Implicit bias and generalization
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase depth from one layer to depth 2 or 3 | Linear regression on the UCI Gas Sensor Array Drift Ethanol task · Ethanol, ℓ2 loss | training convergence speed | Hurt |
| Increase depth from one layer to depth 2 or 3 | Linear regression on the UCI Gas Sensor Array Drift Ethanol task · Ethanol, ℓ4 loss | training convergence speed | Helped |
| Use a depth-3 linear network instead of single-layer AdaGrad or AdaDelta | Linear regression on the UCI Gas Sensor Array Drift Ethanol task · Ethanol, ℓ4 loss | training convergence speed | Helped |
| Use a depth-3 linear network instead of single-layer Adam | Adam optimization on the UCI Gas Sensor Array Drift Ethanol task · Ethanol, ℓ4 loss | training convergence speed | Helped |
| Replace dense-layer matrices with depth-2 linear networks | TensorFlow convolutional network tutorial · MNIST | training convergence speed | Helped |
| Set hidden-layer width to 1 rather than 100 (ablation) | Depth-2 and depth-3 linear networks · Ethanol, ℓ2 and ℓ4 losses | training convergence | No clear effect |
| Use near-zero initialization instead of near-identity initialization (ablation) | Depth-4 and depth-8 linear networks · Ethanol, ℓ4 loss | training convergence speed | Hurt |
| Use near-identity initialization instead of near-zero initialization (ablation) | Depth-4 and depth-8 linear networks · Ethanol, ℓ4 loss | training convergence speed | Helped |
| Increase depth under near-identity initialization (ablation) | Linear networks · Ethanol, ℓ4 loss | training convergence speed | Helped |
| Use depth-2 or depth-3 linear networks instead of a single-layer model (ablation) | Linear regression on the UCI Gas Sensor Array Drift Ethanol task · Ethanol, ℓ2 and ℓ4 losses | training objective trajectory | No clear effect |
Closest papers by meaning
- Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks (NeurIPS 2022 · similarity 0.90)
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization (ICLR 2020 · similarity 0.88)
- Diving into the shallows: a computational perspective on large-scale shallow learning (NeurIPS 2017 · similarity 0.88)
- Is Deeper Better only when Shallow is Good? (NeurIPS 2019 · similarity 0.88)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.88)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.