Research map / Label-efficient and robust learning / Recognition and long-tail
A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn and improve hidden features using gradient descent on structured inputs | Two-layer network · Synthetic learning problems with input structure | hinge loss: ≤ \epsilon | Helped |
| Use fixed data-independent features instead of learned features | Linear models on fixed features · Synthetic learning problems | hinge loss: \pn \left(1 - \frac{\sqrt{2N} B}{2^k}\right) | Hurt |
| Remove the input structure and use uniform patterns | Learning algorithms in the Statistical Query model · Synthetic learning problems without input structure | classification error: less than \frac{1}{2}- \frac{1}{{D \choose k}^3} | Hurt |
| Use only features after the first gradient step, without the second-step improvement (ablation) | Two-layer network · Synthetic data | test accuracy: about 52\% -> about 100\% | Hurt |
| Add a second gradient step to improve the features (ablation) | Two-layer network · Synthetic data | test accuracy: about 52\% -> about 100\% | Helped |
| Mix Gaussian inputs into the training data | Student network trained on CIFAR10 inputs and teacher labels · CIFAR10 and Gaussian mixture | test accuracy | Hurt |
| Mix Gaussian inputs into the training data | Student network trained on MNIST inputs and teacher labels · MNIST and Gaussian mixture | test accuracy | No clear effect |
Closest papers by meaning
- How does Gradient Descent Learn Features --- A Local Analysis for Regularized Two-Layer Neural Networks (NeurIPS 2024 · similarity 0.91)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.89)
- A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks (ICML 2024 · similarity 0.88)
- Complexity Matters: Feature Learning in the Presence of Spurious Correlations (ICML 2024 · similarity 0.87)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.87)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.