Research map / Optimization / Implicit bias and generalization
Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics
Binghua Li, Mengzhe Li, Denny Wu, Tianhao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of population gradient flow for a two-neuron ReLU teacher-student model with isotropic Gaussian inputs, characterizing multi-phase dynamics from small random initialization.
Problem
How gradient descent learns features in a minimal multi-index two-neuron ReLU teacher-student setting, which exhibits rich multi-phase dynamics.
Approach
Analyzes the global loss landscape and bifurcation structure of population gradient flow for a well-specified two-neuron ReLU student learning a two-neuron teacher under Gaussian inputs, showing the dynamics are organized by bifurcations.
Closest papers by meaning
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.90)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.89)
- The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents (ICML 2024 · similarity 0.89)
- Curl Descent : Non-Gradient Learning Dynamics with Sign-Diverse Plasticity (NeurIPS 2025 · similarity 0.89)
- Flat Channels to Infinity in Neural Loss Landscapes (NeurIPS 2025 · similarity 0.88)
- Topological obstruction to the training of shallow ReLU neural networks (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
