Research map / Optimization / Implicit bias and generalization
Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize every layer to the identity and use gradient descent when the initial excess loss is sufficiently small | Deep linear networks on isotropic linear regression · Targets near identity | iterations to excess loss ≤ ε | Helped |
| Use identity-initialized gradient descent for a symmetric positive definite target | Deep linear networks · Symmetric γ-positive targets | iterations to excess loss ≤ ε | Helped |
| Add power projection onto the γ-positive set and balanced factorization after gradient steps | Deep linear networks · γ-positive targets, including nonsymmetric targets | iterations to excess loss ≤ ε | Helped |
| Use identity-initialized gradient methods on a symmetric target with a negative eigenvalue and even depth | Penalty-regularized gradient descent · Symmetric targets with a negative eigenvalue | excess loss: ℓ(Θ^(t)) ≥ λ²/2 | Hurt |
| Project layers into a bounded neighborhood of the identity | ψ-step-and-project algorithms · Symmetric targets with a negative eigenvalue | excess loss: ℓ(Θ^(t)) ≥ λ²/2 | Hurt |
| Regularize each layer toward the identity | Penalty-regularized gradient descent · Symmetric targets with a negative eigenvalue and even depth | ability to minimize loss: ℓ(Θ^(t)) ≥ λ²/2 | Hurt |
| Use early stopping or a per-layer identity penalty for a symmetric target with a negative eigenvalue | Identity-initialized gradient methods · Symmetric targets with a negative eigenvalue | ability to converge to the optimal solution | Hurt |
Closest papers by meaning
- Global Convergence of Gradient Descent for Deep Linear Residual Networks (NeurIPS 2019 · similarity 0.91)
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (ICLR 2019 · similarity 0.90)
- Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced (NeurIPS 2018 · similarity 0.90)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.90)
- How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization (ICLR 2024 · similarity 0.88)
- Provable Acceleration of Nesterov's Accelerated Gradient for Asymmetric Matrix Factorization and Linear Neural Networks (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.