Research map / Optimization / Implicit bias and generalization
Global Convergence of Gradient Descent for Deep Linear Residual Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZAS initialization instead of standard Xavier or near-identity initialization | Gradient descent for one-dimensional deep linear networks · Target Φ = -I | iterations to convergence: \exp(\Omega(L)) versus O\left( L^3 \log(1/\varepsilon) \right) | Helped |
| Use ZAS instead of near-identity initialization | Gradient descent for a 6-layer linear network · Negative identity matrix | saddle-region escape: 2 of 5 | Helped |
| Use mZAS instead of Xavier initialization | Gradient descent for nonlinear residual networks · Fashion-MNIST (1000 training samples) | convergence | Helped |
| Use mZAS instead of Xavier initialization | Gradient descent for nonlinear residual networks · Fashion-MNIST, depth L = 2000, 10000 | optimization stability: 2000, 10000 | Helped |
Closest papers by meaning
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (ICLR 2019 · similarity 0.93)
- On the Global Convergence of Training Deep Linear ResNets (ICLR 2020 · similarity 0.91)
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks (ICLR 2019 · similarity 0.91)
- Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks (ICML 2018 · similarity 0.91)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.