Research map / Optimization / Implicit bias and generalization
Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuron
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, Sham M. Kakade
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GLM-tron instead of SGD | ReLU regression with symmetric Bernoulli data, well-specified setting · Well-specified ReLU regression, symmetric Bernoulli | expected excess risk: min_{\gamma^{\tron}_0<1/2} \Ebb \excessrisk(\wB_N^{\tron}) \lesssim min_{\gamma^{\sgd}_0<1}\Ebb \excessrisk(\wB_N^{\sgd} | Improved |
| Use GLM-tron instead of SGD | Noiseless ReLU regression with symmetric Bernoulli data · Noiseless ReLU regression, symmetric Bernoulli | expected population risk: \Ebb_{\wB_*} \Ebb_{\algo} \risk(\wB_N^{\tron}) \lesssim \big\| \wB_0 - \wB_* \big\|^2_{\prod_{t=1}^{N}\big( \IB-\frac{\g | Improved |
| Apply GLM-tron in the well-specified setting under symmetric Bernoulli features | GLM-tron · Well-specified ReLU regression, symmetric Bernoulli | expected excess risk: \Ebb \excessrisk (\wB_{N}) \lesssim \big\| \wB_0 - \wB_* \big\|^2_{\prod_{t=1}^{N}(\IB-\frac{\gamma_t}{2}\HB)\HB} + \sig | Improved |
| Apply GLM-tron in the misspecified setting with hypercontractive features | GLM-tron · Misspecified ReLU regression | expected population risk: \Ebb \risk (\wB_{N}) \lesssim \OPT + \bigg\| \prod_{t=1}^{N}\Big(\IB-\frac{\gamma_t}{2}\HB\Big)(\wB_0 - \wB_*) \bigg\|^2 | Improved |
| Use GLM-tron's finite-dimensional bound rather than its prior \Ocal(\OPT + \sqrt{d/N}) bound | GLM-tron · Misspecified ReLU regression, finite dimension | expected population risk: \OPT + \frac{d}{\Neff} = \OPT + \frac{d\log(N)}{N}; prior bound: \Ocal(\OPT + \sqrt{d/N}) | Improved |
| Replace GLM-tron's update with the ReLU-derivative update used by SGD | GLM-tron on noiseless ReLU regression with symmetric Bernoulli data · Noiseless ReLU regression, symmetric Bernoulli | expected population risk: \Ebb_{\wB_*} \Ebb_{\algo} \risk(\wB_N^{\sgd}) \ge \half \cdot \| \wB_*\|^2_{\HB}\ge \half \cdot \risk(0) | Hurt |
Closest papers by meaning
- Agnostic Learning of General ReLU Activation Using Gradient Descent (ICLR 2023 · similarity 0.88)
- Agnostic Learning of a Single Neuron with Gradient Descent (NeurIPS 2020 · similarity 0.88)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.88)
- Learning a Single Neuron with Bias Using Gradient Descent (NeurIPS 2021 · similarity 0.88)
- Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noise (NeurIPS 2024 · similarity 0.87)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
