Research map / Optimization / Implicit bias and generalization
Learning a Single Neuron with Bias Using Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a bias initialized from U([-1,1]) along with the other coordinates | Gradient descent for a single ReLU neuron · Uniform ball around the origin, radius ε | probability of getting stuck at initialization: > 1/2 - ε√d | Hurt |
| Use a target with a sufficiently negative bias, even when the learned bias starts at zero | Gradient flow for a single ReLU neuron · Uniform ball of radius r | probability of not reaching the global minimum: at least 1/2 - o_d(1) | Hurt |
| Require initial loss to be at least δ below F(0), with the stated small step size | Gradient descent for a single ReLU neuron with bias · Distributions satisfying Assumption 4.2 | squared distance to target: ‖w_t - v‖² ≤ ‖w₀ - v‖² (1 - γη)^t | Helped |
| Use small spherical random initialization with zero bias under the target-bias condition | Gradient descent for a single ReLU neuron with bias · Distributions satisfying the two random-initialization assumptions | probability of linear convergence to the global minimum: at least 1-e^{Ω(d)} | Helped |
| Initialize within unit squared distance of the target with nonnegative bias | Gradient descent for a single ReLU neuron with bias · Spherically symmetric distributions satisfying the stated assumptions | squared distance to target: ‖w_t - v‖² ≤ (1-ηλ)^t‖w₀ - v‖² | Helped |
| Add an explicit bias to the single-neuron learning problem | Gradient methods for a single ReLU neuron · Uniform distribution over a ball | probability of convergence to the global minimum: without a bias term ... probability 1; with bias ... probability close to half | Hurt |
Closest papers by meaning
- Agnostic Learning of a Single Neuron with Gradient Descent (NeurIPS 2020 · similarity 0.91)
- Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs (NeurIPS 2023 · similarity 0.90)
- Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noise (NeurIPS 2024 · similarity 0.88)
- On the Feasibility of Single-Pass Full-Capacity Learning in Linear Threshold Neurons with Binary Input Vectors (ICML 2024 · similarity 0.86)
- Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network (ICML 2024 · similarity 0.85)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.