Research map / Optimization / Implicit bias and generalization
A Convergence Theory for Deep Learning via Over-Parameterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Over-parameterize a deep ReLU network and train its hidden layers from random initialization with gradient descent | L-layer fully connected ReLU network · Non-degenerate training data, ℓ2 regression | iterations to reach F(W) ≤ ε: T = Θ( (poly(n, L)/δ²) · log ε⁻¹ ) | Helped |
| Use mini-batch SGD on an over-parameterized deep ReLU network | L-layer fully connected ReLU network · Non-degenerate training data, ℓ2 regression | iterations to reach F(W) ≤ ε: T = Θ( (poly(n, L) · log² m)/(δ² b) · log ε⁻¹ ) | Helped |
| Use sufficient polynomial over-parameterization | Deep neural networks trained from random initialization · Non-degenerate training data | training accuracy: 100% | Helped |
| Extend the over-parameterized training analysis to a convolutional architecture | CNN · Non-degenerate training data, ℓ2 regression | iterations to reach F(W) ≤ ε: T = poly(n, L, d_in)/δ² log 1/ε | Helped |
| Extend the over-parameterized training analysis to a convolutional architecture with mini-batch SGD | CNN · Non-degenerate training data, ℓ2 regression | iterations to reach F(W) ≤ ε: T = poly(n, L, d_in) · log² m/δ² log 1/ε | Helped |
Closest papers by meaning
- An Improved Analysis of Training Over-parameterized Deep Neural Networks (NeurIPS 2019 · similarity 0.93)
- On the Global Convergence of Training Deep Linear ResNets (ICLR 2020 · similarity 0.91)
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (ICLR 2019 · similarity 0.91)
- Gradient Descent Finds Global Minima of Deep Neural Networks (ICML 2019 · similarity 0.91)
- On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths (ICML 2021 · similarity 0.91)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.