Research map / Optimization / Implicit bias and generalization
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train first-layer weights with online SGD and weight decay | Two-layer neural network on a multiple-index teacher model · Multiple-index teacher model | distance to principal subspace: \frac{\|W^T_\perp\|_F}{\sqrt{m}} \lesssim \sqrt{\frac{d + \log(1/\delta)}{\gamma^2T}} | Helped |
| Use a ReLU student trained with SGD to learn a monotone single-index target | ReLU two-layer network · Single-index model | excess risk: \lesssim \Delta_*^2\left\{\sqrt{\frac{\log(T/\delta)}{m}} + \sqrt{\frac{d + \log(1/\delta)}{T}}\right\} + \nu\sqrt{\frac | Helped |
| Apply uniform convergence on the learned principal subspace | SGD-trained two-layer neural network · Generalization bound, multiple-index model | generalization gap: \lesssim \tau r_a\left\{\sqrt{\frac{\kappa(d + \log(1/\delta))}{\gamma^2T}} + (r_b + {\tilde{\lambda}}^{-1})\sqrt{\frac{ | Helped |
| Compress the trained first layer to rank k | SGD-trained two-layer neural network · Compressed rank-k network | risk deviation: \lesssim \frac{\tau\kappa}{\gamma}\sqrt{\frac{d + \log(1/\delta)}{T}} | Helped |
| Remove weight decay (ablation) | SGD-trained two-layer neural network · Principal-subspace convergence | convergence to principal subspace | Hurt |
| Set the weight-decay parameter to zero (ablation) | Population gradient descent on a two-layer neural network · Principal-subspace convergence | convergence to principal subspace | Hurt |
Closest papers by meaning
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.91)
- Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit (NeurIPS 2024 · similarity 0.90)
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.90)
- SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem (ICLR 2024 · similarity 0.89)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.88)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.