Research map / Optimization / Implicit bias and generalization
On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a sufficiently over-parameterized student (M=100 rather than M=5) | Norm-dependent gradient descent · Teacher-student regression, m=5, d=5, n=100 | training loss | Helped |
| Use a sufficiently over-parameterized student (M=100 rather than M=5 or M=10) | Norm-dependent gradient descent · Teacher-student regression, m=5, d=5, n=100 | test loss | Helped |
| Replace sparse L1 regularization with L2 regularization (ablation) | Norm-dependent gradient descent · Teacher-student regression, m=5, d=5, n=100, M=10 | loss trajectory | No clear effect |
| Apply sparse regularization so nodes far from teacher parameters are eliminated | Two-dimensional teacher-student regression · d=2, m=2, M=15, n=100 | node selection | Helped |
| Add sparse regularization with sufficiently small λ | Regularized empirical-risk minimization · Teacher-student regression | squared parameter error: O(mλ^2) | Helped |
| Use norm-dependent gradient descent with sufficient over-parameterization | Regularized two-layer ReLU student · Teacher-student regression | regularized objective: J(ν_k)-J*≤(J(ν_0)-J*)(1-κ_0)^{k-k_0} | Helped |
Closest papers by meaning
- Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods (ICLR 2023 · similarity 0.95)
- Student Specialization in Deep Rectified Networks With Finite Width and Input Dimension (ICML 2020 · similarity 0.91)
- Convex Formulations for Training Two-Layer ReLU Neural Networks (ICLR 2025 · similarity 0.87)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.86)
- Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes (ICML 2024 · similarity 0.86)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.