Research map / Optimization / Implicit bias and generalization

Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Gradient flow and mean-field dynamics

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretical analysis of SGD dynamics for two-layer networks on a multi-index teacher, decomposing first-layer weights into teacher-aligned signal and orthogonal components to study their effect on generalization.

Problem

How first-layer parameters orthogonal to the teacher subspace affect generalization of two-layer networks trained by SGD on a multi-index teacher model.

Approach

Decomposes first-layer weights into a teacher-aligned signal component and an orthogonal complement, and analyzes their joint SGD dynamics with the second layer in the proportional limit.

Surprising findings

• Teacher-induced spikes and variance inflation arise from first-layer parameters orthogonal to the teacher subspace (as indicated by title).

Experimental setup

Metrics: generalization error

Closest papers by meaning

• Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.90)
• Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.89)
• Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.89)
• Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks (NeurIPS 2025 · similarity 0.88)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.88)
• SGD vs GD: Rank Deficiency in Linear Networks (NeurIPS 2024 · similarity 0.87)