Research map / Optimization / Implicit bias and generalization

Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Implicit regularization of SGD · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Random-matrix analysis of gradient flow in a linear teacher-student model with two-block input covariance, deriving a 2x2 Dyson equation for the symmetrized weight matrix and a rank-two outlier condition that yields a transient BBP transition enabling early stopping.

Problem

Explaining why a signal-carrying outlier eigenvalue of trained weight matrices appears transiently during gradient descent and is then reabsorbed into the bulk, i.e., a mechanistic account of early stopping in a solvable spectral model.

Approach

Uses anisotropic (two-block) input covariance so that the symmetrized gradient-flow weight matrix is a two-block Wigner-type matrix with time-dependent variance profile; the finite-rank teacher perturbation is rank two, and an outlier condition is derived via a 2x2 determinant, giving a transient BBP transition.

Key results

• Two-block 2x2 Dyson equation predicts the time-dependent bulk spectrum of the symmetrized weight matrix, matching finite-size simulations (N=500, 20 realizations) at eight training times.
• Three regimes (weak-signal, strong-signal, early-stopping transient window) are identified; the transient window opens only below a critical anisotropy threshold lambda_-*, per the (theta, lambda_-) phase diagram.
• In the power-law full-rank spectrum experiment (beta=1.5, N=400), the teacher overlap q_t rises, peaks, and returns to baseline 1/N for intermediate theta.

Surprising findings

• For nearly isotropic inputs the transient early-stopping regime shrinks and disappears, so early stopping is only useful for sufficiently anisotropic data (per the paper's claim).

Limitations

• Symmetrized matrix S_t = A + A^T is used instead of the standard A A^T, so results do not directly transfer to Wishart-type spectra.
• Linear teacher-student model; does not capture heavy-tailed bulk or nonlinear multi-layer coupling of deep networks.
• Optimal stopping time and test-error link are only numerically identified; rigorous test-error decomposition left to future work.

Experimental setup

Datasets: synthetic Gaussian linear teacher-student data

Metrics: empirical spectral density agreement with Dyson prediction outlier location teacher-eigenvector overlap q_t phase diagram classification

Builds on: Baik-Ben Arous-Péché BBP transition Benaych-Georges and Nadakuditi finite-rank perturbation Dyson equation for Wigner-type matrices with variance profile

Closest papers by meaning

• Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models (ICML 2019 · similarity 0.88)
• Spectral Phase Transition and Optimal PCA in Block-Structured Spiked Models (ICML 2024 · similarity 0.87)
• Free Decompression with Algebraic Spectral Curves (NeurIPS 2026 · similarity 0.87)
• Optimal Spectral Transitions in High-Dimensional Multi-Index Models (NeurIPS 2025 · similarity 0.86)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.86)
• An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models (NeurIPS 2025 · similarity 0.85)