Research map / Optimization / Implicit bias and generalization
A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use balanced initialization instead of layer-wise independent Gaussian initialization | 3-layer linear network · Gas Sensor Array Drift (Ethanol) | convergence time | Helped |
| Use balanced initialization instead of layer-wise independent Gaussian initialization | 8-layer linear network · Gas Sensor Array Drift (Ethanol) | convergence time | Helped |
| Increase depth from 3 layers to 8 layers (ablation) | Layer-wise independently initialized linear network · Gas Sensor Array Drift (Ethanol) | convergence time | Hurt |
| Vary the standard deviation of layer-wise independent Gaussian initialization (ablation) | Linear networks · Gas Sensor Array Drift (Ethanol) | convergence time | Mixed |
| Replace layer-wise independent initialization with balanced initialization | Fully connected ReLU network · MNIST | training loss after 10 epochs | Helped |
Closest papers by meaning
- Global Convergence of Gradient Descent for Deep Linear Residual Networks (NeurIPS 2019 · similarity 0.93)
- On the Global Convergence of Training Deep Linear ResNets (ICLR 2020 · similarity 0.93)
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks (ICLR 2019 · similarity 0.93)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.90)
- Block Coordinate Descent for Neural Networks Provably Finds Global Minima (NeurIPS 2025 · similarity 0.89)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.