AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

On the Convergence of Gradient Flow on Multi-layer Linear Models

Hancheng Min, Rene Vidal, Enrique Mallada

ICML 2023 · Optimization › Implicit bias and generalization · approach: Gradient flow and mean-field dynamics

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Initialize weights with non-zero imbalance (imbalance matrices bounded away from zero, Delta > 0) instead of balanced initializationtwo-layer linear models with gradient flow · theoretical bound (gradient flow, PL loss)convergence rate lower bound lambda_min(T)Improved
Add a margin-based bound on the least singular value of the weight product, controlled by initializing the product close to the optimumtwo-layer linear models with f satisfying A1 and A2 · theoretical bound (gradient flow, strongly convex smooth f)convergence rate lower bound alpha_2Improved
Use unimodal weight initialization (ordering of imbalance matrices w.r.t. PSD cone) for deep linear networksdeep linear models, depth L · theoretical bound (gradient flow)convergence rate lower bound alpha_LImproved
Random Gaussian initialization (NTK, Xavier, Fanout) outside kernel regime on three-layer networksthree-layer linear network · numerical, bound Delta* over 100 random samplespositivity of bound Delta*(D21,D23)Improved
Theoretical bound used as predictor of actual convergence rate in three-layer networks with larger dimension (n=5, m=1)three-layer linear network under Fanout init · gradient descent loss curve vs boundtightness of convergence-rate boundMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy