Research map / Optimization / Implicit bias and generalization
How Gradient descent balances features: A dynamical analysis for two-layer neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical dynamical analysis of gradient descent for training a width-m two-layer ReLU student network to learn k teacher neurons from Gaussian inputs, showing a three-phase trajectory (alignment after weak recovery, tangential growth, and a final phase).
Problem
Understanding whether and how gradient descent on two-layer ReLU networks learns k teacher neurons from Gaussian data, i.e., how features are balanced among student neurons during training.
Approach
Analyzes gradient descent dynamics with proper initialization and small step size in phases (alignment after weak recovery, tangential growth, and a further phase), giving a dynamical account of how student neurons balance features across teachers.
Closest papers by meaning
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (NeurIPS 2019 · similarity 0.92)
- Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks (ICML 2026 · similarity 0.92)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.89)
- The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents (ICML 2024 · similarity 0.89)
- Improving Deep Learning Speed and Performance Through Synaptic Neural Balance (AAAI 2025 · similarity 0.88)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.