Research map / Optimization / Implicit bias and generalization
Gradient descent aligns the layers of deep linear networks
Ziwei Ji, Matus Telgarsky
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply gradient flow to a deep linear network on linearly separable data | Deep linear network · Linearly separable data | risk: risk converges to 0 | Improved |
| Apply gradient flow to a deep linear network on linearly separable data | Deep linear network · Linearly separable data | normalized weight matrix rank-one alignment: W_k/||W_k||_F converges to u_kv_k^T | Improved |
| Apply gradient flow to a deep linear network on linearly separable data | Deep linear network · Linearly separable data | adjacent-layer alignment: |v_{k+1}^T u_k| -> 1 | Improved |
| Use logistic loss with gradient flow | Deep linear network · Linearly separable data | predictor alignment with maximum-margin solution: wnn/||wnn|| -> u-bar | Improved |
| Use adaptive decreasing step sizes for gradient descent | Deep linear network · Linearly separable data | risk: risk converges to 0 | Improved |
| Use adaptive decreasing step sizes for gradient descent | Deep linear network · Linearly separable data | normalized network predictor alignment: normalized network predictor aligns with v_1 | Improved |
| Use logistic loss with gradient descent | Deep linear network · Linearly separable data | predictor alignment with maximum-margin solution: wnn/∏||W_k||_F -> u-bar | Improved |
Closest papers by meaning
- On the Convergence of Gradient Flow on Multi-layer Linear Models (ICML 2023 · similarity 0.90)
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (ICLR 2019 · similarity 0.90)
- Directional convergence and alignment in deep learning (NeurIPS 2020 · similarity 0.90)
- Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced (NeurIPS 2018 · similarity 0.89)
- Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.87)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
