Research map / Optimization / Implicit bias and generalization
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks
Eshaan Nichani, Alex Damian, Jason D. Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a three-layer network with layer-wise training instead of a two-layer network or kernel method | Algorithm 1 · f*(x)=ReLU(x^T A x), d=32 | test loss vs sample size | Improved |
| learn quadratic feature targets g*(x^T A x) with a three-layer network instead of kernel/NTK two-layer methods | Algorithm 1 · functions of quadratic features | sample complexity: d^4 vs d^2p | Improved |
| apply Algorithm 1 to single-index models with sigma2(z)=z | Algorithm 1 · single-index models | sample complexity | No clear effect |
| restrict link function g* to a degree p=O(1) polynomial | Algorithm 1 · g*(x^T A x) | error floor | Improved |
| use a two-layer network instead of Algorithm 1 on ReLU(x^T A x) | two-layer network · depth separation target | width or weight norm required | Hurt |
| train layer-wise with Algorithm 1 on smoothed ReLU target | Algorithm 1 · ReLU(x^T A x)-c0 | population L2 loss bound: d^-1/6 error floor | Improved |
Closest papers by meaning
- A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks (ICML 2024 · similarity 0.87)
- Learning Hierarchical Polynomials with Three-Layer Neural Networks (ICLR 2024 · similarity 0.87)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.86)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.86)
- How does Gradient Descent Learn Features --- A Local Analysis for Regularized Two-Layer Neural Networks (NeurIPS 2024 · similarity 0.86)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
