Research map / Optimization / Implicit bias and generalization

Learning Dynamics of Deep Matrix Factorization Beyond the Edge of Stability

ICLR 2025 · Optimization › Implicit bias and generalization · approach: Explicit and implicit regularization interplay

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes gradient descent dynamics of deep matrix factorization (deep linear networks) beyond the edge of stability, characterizing how the sharpness and loss evolve under a fixed learning rate.

Problem

Deep networks trained with gradient descent at a fixed learning rate often operate at the edge of stability, where the top Hessian eigenvalue hovers near 2/η; the training dynamics in this regime are poorly understood, especially for deep linear networks.

Approach

Fine-grained theoretical analysis of deep matrix factorization loss beyond the edge of stability, tracking how loss oscillates and sharpness equilibrates for deep linear networks, rather than only the sharpness threshold.

Experimental setup

Builds on: edge of stability

Closest papers by meaning

• Understanding Edge-of-Stability Training Dynamics with a Minimalist Example (ICLR 2023 · similarity 0.91)
• Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.90)
• Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability (NeurIPS 2025 · similarity 0.89)
• Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.88)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.88)
• Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario (NeurIPS 2026 · similarity 0.88)