Research map / Optimization / Implicit bias and generalization
On the Convergence of Gradient Flow on Multi-layer Linear Models
Hancheng Min, Rene Vidal, Enrique Mallada
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize weights with non-zero imbalance (imbalance matrices bounded away from zero, Delta > 0) instead of balanced initialization | two-layer linear models with gradient flow · theoretical bound (gradient flow, PL loss) | convergence rate lower bound lambda_min(T) | Improved |
| Add a margin-based bound on the least singular value of the weight product, controlled by initializing the product close to the optimum | two-layer linear models with f satisfying A1 and A2 · theoretical bound (gradient flow, strongly convex smooth f) | convergence rate lower bound alpha_2 | Improved |
| Use unimodal weight initialization (ordering of imbalance matrices w.r.t. PSD cone) for deep linear networks | deep linear models, depth L · theoretical bound (gradient flow) | convergence rate lower bound alpha_L | Improved |
| Random Gaussian initialization (NTK, Xavier, Fanout) outside kernel regime on three-layer networks | three-layer linear network · numerical, bound Delta* over 100 random samples | positivity of bound Delta*(D21,D23) | Improved |
| Theoretical bound used as predictor of actual convergence rate in three-layer networks with larger dimension (n=5, m=1) | three-layer linear network under Fanout init · gradient descent loss curve vs bound | tightness of convergence-rate bound | Mixed |
Closest papers by meaning
- Understanding the Dynamics of Gradient Flow in Overparameterized Linear models (ICML 2021 · similarity 0.91)
- Gradient descent aligns the layers of deep linear networks (ICLR 2019 · similarity 0.90)
- Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization (ICML 2026 · similarity 0.90)
- On the spectral bias of two-layer linear networks (NeurIPS 2023 · similarity 0.89)
- SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures (NeurIPS 2025 · similarity 0.87)
- Transformative or Conservative? Conservation laws for ResNets and Transformers (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
