Research map / Optimization / Implicit bias and generalization
Gradient Dynamics of Shallow Univariate ReLU Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use kernel dynamics rather than adaptive dynamics | Shallow univariate ReLU networks · Least-squares interpolation | interpolant smoothness | Helped |
| Use adaptive dynamics rather than kernel dynamics | Shallow univariate ReLU networks · Least-squares interpolation | knot adaptivity | Helped |
| Initialize with second-layer weights much larger than first-layer weights | Full-parameter gradient flow · Univariate least-squares interpolation | adaptive learning | Helped |
| Initialize with first-layer weights much larger than second-layer weights | Full-parameter gradient flow · Univariate least-squares interpolation | kernel behavior | Helped |
| Use normalization α(m)=o(m) | Full-parameter gradient flow · Infinite-width limit | lazy behavior | Helped |
| Use uniform initialization over intervals for the random-feature kernel | Random-feature kernel · Univariate interpolation | cubic-spline form | Helped |
| Use a radial initialization such as jointly Gaussian weights instead of interval-uniform initialization | Random-feature kernel · Univariate interpolation | exact cubic-spline form | Hurt |
| Train only the output weights, keeping first-layer parameters fixed (ablation) | Shallow univariate ReLU networks · Square-wave fit | knot adaptivity | Hurt |
| Train the first-layer parameters rather than only the output weights (ablation) | Shallow univariate ReLU networks · Square-wave fit | knot adaptivity | Helped |
| Reduce the neuron count under the described default initialization (ablation) | Shallow univariate ReLU networks · Sine-wave fit | adaptivity | Helped |
Closest papers by meaning
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.89)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.89)
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.89)
- Transformative or Conservative? Conservation laws for ResNets and Transformers (ICML 2025 · similarity 0.89)
- Topological obstruction to the training of shallow ReLU neural networks (NeurIPS 2024 · similarity 0.89)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.