Research map / Learning theory and algorithms / Prediction and decision losses
Towards Understanding Learning in Neural Networks with Linear Teachers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use small initialization scale | Two-layer Leaky ReLU network trained on linearly separable data · Synthetic and real datasets | decision-boundary linearity | Helped |
| Cluster neurons with the same output-weight sign | Two-layer Leaky ReLU network · Linearly separable data | decision-boundary linearity | Helped |
| Train with SGD using cross-entropy | Two-layer Leaky ReLU network with fixed second-layer signs · Linearly separable training set | empirical cross-entropy loss: O\left(\frac{n^4}{\varepsilon^2}\right) | Helped |
| Keep the gradient-flow trajectory in a Neural Agreement Regime | Two-layer Leaky ReLU network · Linearly separable training set | neuron clustering | Helped |
| Use Leaky ReLU rather than ReLU | Clustered two-layer network · Linearly separable data | decision-boundary linearity | Helped |
Closest papers by meaning
- SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data (ICLR 2018 · similarity 0.90)
- The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions (ICLR 2022 · similarity 0.88)
- A Convergence Theory for Deep Learning via Over-Parameterization (ICML 2019 · similarity 0.88)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.86)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.86)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.