Research map / Learning theory and algorithms / Sample complexity
Pareto Frontiers in Deep Feature Learning: Data, Compute, Width, and Luck
Benjamin L. Edelman, Surbhi Goel, Sham M. Kakade, eran malach, Cyril Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase MLP width up to 10^5 | 2-layer MLP · Offline sparse parity | success probability | Improved |
| increase MLP width | 2-layer MLP · Offline sparse parity | sample efficiency | Improved |
| increase MLP width | 2-layer MLP · Offline sparse parity | iterations to convergence | Improved |
| sparse axis-aligned (2-hot) initialization instead of dense initialization (ablation) | 2-layer MLP · Offline sparse parity | success probability and sample efficiency | Improved |
| initialize each neuron with s=2 hot weights versus other sparsity levels (ablation) | Sparse init MLP · Offline sparse parity | success probability | Improved |
| reduce dataset size toward the edge of feasibility | 2-layer MLP · Offline sparse parity | iterations to convergence | Hurt |
| increase dataset size m | 2-layer MLP · Offline sparse parity | success probability | Mixed |
| increase MLP width on downsampled tabular data | MLP · OpenML tabular benchmark (16 tasks) | test error | Improved |
| sparse axis-aligned initialization on tabular data (ablation) | MLP · OpenML tabular benchmark (16 tasks) | test error | Mixed |
| wide sparse-init MLP versus tuned random forest | MLP · OpenML tabular benchmark | test error | Mixed |
Closest papers by meaning
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.86)
- No Free Prune: Information-Theoretic Barriers to Pruning at Initialization (ICML 2024 · similarity 0.86)
- Explainable Neural Networks with Guarantee: A Sparse Estimation Approach (AAAI 2025 · similarity 0.86)
- Matching the Statistical Query Lower Bound for $k$-Sparse Parity Problems with Sign Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.86)
- Position: A Theory of Deep Learning Must Include Compositional Sparsity (ICML 2025 · similarity 0.85)
- Complexity Matters: Feature Learning in the Presence of Spurious Correlations (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
