Research map / Optimization / Implicit bias and generalization
On the Global Convergence of Training Deep Linear ResNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gaussian random input and output transformations with zero-initialized hidden weights | Deep linear ResNet trained by GD · Linear regression training loss | iterations to reach ε-suboptimal training loss: O(κ log(1/ε)) | Helped |
| Use Gaussian random input and output transformations with zero-initialized hidden weights | Deep linear ResNet trained by SGD · Linear regression training loss | iterations to reach ε-suboptimal training loss: \tilde O(κ^2ε^{-1}log(1/ε)·n/B) | Helped |
| Use modified identity transformations with disjoint input/output coordinates (ablation) | Deep linear ResNet trained by GD · 10-d synthetic data | convergence | Helped |
| Use random input/output transformations (ablation) | Deep linear ResNet trained by GD · 10-d synthetic data | convergence | Helped |
| Use identity input/output transformations rather than modified identity or random transformations (ablation) | Deep linear ResNet trained by GD · 10-d synthetic data | convergence | Hurt |
| Use modified identity transformations rather than random transformations (ablation) | Deep linear ResNet trained by GD · 10-d synthetic data | convergence rate | Helped |
| Train a deep linear ResNet rather than a standard deep linear network | Linear networks on synthetic data · Synthetic data, varying depth | convergence performance | Helped |
| Increase depth (ablation) | Deep linear ResNet · Synthetic data, varying depth | convergence performance | No clear effect |
| Increase depth (ablation) | Standard deep linear network · Synthetic data, varying depth | convergence performance | Hurt |
| Use the deep linear ResNet width condition instead of the standard deep linear network width condition | GD training of deep linear networks · Linear regression training loss | required network width: factor of O(Lκ) | Helped |
| Use modified identity input/output transformations with m = d + k | Deep linear ResNet trained by GD · Linear regression training loss | global convergence guarantee: m = d + k | Helped |
| Use identity transformations that copy inputs and select output coordinates (ablation) | Deep linear ResNet trained by GD · Linear regression training loss | global convergence guarantee | Hurt |
Closest papers by meaning
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (ICLR 2019 · similarity 0.93)
- On global convergence of ResNets: From finite to infinite width using linear parameterization (NeurIPS 2022 · similarity 0.91)
- A Convergence Theory for Deep Learning via Over-Parameterization (ICML 2019 · similarity 0.91)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.89)
- Generalization of Scaled Deep ResNets in the Mean-Field Regime (ICLR 2024 · similarity 0.89)
- Block Coordinate Descent for Neural Networks Provably Finds Global Minima (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.