Research map / Optimization / Implicit bias and generalization
Convergence Analysis of Two-layer Neural Networks with ReLU Activation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a one-layer identity mapping to a 56-layer network (ablation) | Vanilla network · CIFAR-10 | test error: 12.04% -> 9.01% | Helped |
| Use a standard ResNet identity mapping instead of a one-layer skip | 56-layer network · CIFAR-10 | test error: 9.01% -> 6.97% | Helped |
| Remove identity mappings (ablation) | 56-layer Single skip network · CIFAR-10 | test error: 9.01% -> 12.04% | Hurt |
| Use zero initialization rather than standard random initialization (ablation) | 5-block network with identity mappings · MNIST | test error: 1.27% -> 1.28% | No clear effect |
| Remove identity mappings while using zero initialization (ablation) | 5-block network · MNIST | test error | Hurt |
| Use weights with spectral norm 0.6 instead of W = 0 | f(x, W) · MNIST, classes 0 and 1 | training error: 7% -> 1% | Helped |
| Use weights with spectral norm 0.6 instead of W = 0 | f(x, W) · MNIST, all 45 class pairs | average training error: 34% -> 14% | Helped |
| Initialize the network in the identity-mapping region rather than use the vanilla initialization (ablation) | Two-layer student network · Gaussian teacher-student experiment | distance to optimal | Helped |
| Use a three-block student instead of a vanilla network | Teacher-student experiment · Gaussian teacher-student experiment | training error | Helped |
| Use a three-block student instead of ResLink | Teacher-student experiment · Gaussian teacher-student experiment | test error | Hurt |
Closest papers by meaning
- Neural Networks Efficiently Learn Low-Dimensional Representations with SGD (ICLR 2023 · similarity 0.91)
- The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions (ICLR 2022 · similarity 0.90)
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks (ICLR 2019 · similarity 0.90)
- SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem (ICLR 2024 · similarity 0.88)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.88)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.