Research map / Learning theory and algorithms / Prediction and decision losses
Small nonlinearities in activation functions create bad local minima in neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a ReLU-like piecewise-linear nonnegative homogeneous activation instead of a linear model | One-hidden-layer network with squared loss and hidden width at least 2 · Datasets not linearly fittable | empirical risk / existence of spurious local minima | Hurt |
| Introduce even a slight nonlinearity, with s_+ = 1+ε and s_- = 1 | One-hidden-layer network · Datasets not linearly fittable | empirical risk / existence of spurious local minima | Hurt |
| Use an activation satisfying the theorem’s conditions, including sigmoid, tanh, arctan, ELU, SELU, or ReLU | One-hidden-layer network on the synthetic 3-point dataset · Synthetic 3-point dataset | squared empirical risk: \frac{1}{3} vs. 0 | Mixed |
| Reparameterize a differentiable loss as a deep linear network, with hidden widths at least min{d_x,d_y} and all critical points of the base loss globally optimal | Deep linear network · General differentiable loss under stated assumptions | spurious local minima | Helped |
Closest papers by meaning
- Piecewise linear activations substantially shape the loss surfaces of neural networks (ICLR 2020 · similarity 0.92)
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks (ICML 2018 · similarity 0.92)
- The Loss Surface of Deep and Wide Neural Networks (ICML 2017 · similarity 0.91)
- Deep Linear Networks with Arbitrary Loss: All Local Minima Are Global (ICML 2018 · similarity 0.90)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.