Research map / Optimization / Implicit bias and generalization
Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a same-width ReLU student with two-phase gradient descent | Two-layer ReLU student · Teacher-student ReLU regression | excess risk: O(\samplesize^{-1}) versus \gtrsim \samplesize^{-\frac{\Dim+2}{2\Dim+2}}\samplesize^{-\kappa} | Helped |
| Use noisy gradient descent in phase I to explore for teacher-neuron parameters, followed by vanilla gradient descent in phase II | Two-layer ReLU student · Teacher-student ReLU regression | excess risk: O(\samplesize^{-1}) | Helped |
| Use vanilla gradient descent instead of noisy gradient descent for the global-search phase | Teacher-student ReLU network optimization · Teacher-student ReLU regression | computational complexity to find the local region: exponentially large width | Hurt |
Closest papers by meaning
- On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting (ICML 2021 · similarity 0.95)
- Student Specialization in Deep Rectified Networks With Finite Width and Input Dimension (ICML 2020 · similarity 0.91)
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (NeurIPS 2019 · similarity 0.89)
- Convex Formulations for Training Two-Layer ReLU Neural Networks (ICLR 2025 · similarity 0.87)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.