Research map / Optimization / Implicit bias and generalization
Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that infinite-width L-layer MLPs trained with SGD under μP (with mild smooth, GOOD activations) keep features linearly independent while substantially learning features, so any convergent point is a global minimum.
Problem
Whether deep networks can simultaneously learn meaningful features (rich feature learning) and provably converge to a global minimum, since NTK-regime analyses keep features near initialization and mean-field deep analyses suffer feature collapse.
Approach
Uses the tensor program framework under μP and a Gaussian process view with covariance invariance across adjacent layers and a filtration-based induction to prove that pre- and post-activation features remain linearly independent throughout SGD training, so convergence implies vanishing error signals and hence global optimality.
Key results
- Under μP, feature Gram matrices stay non-degenerate with minimum eigenvalue that does not collapse as width grows, while IP shows feature collapse and NTP/SP eigenvalues decay with width.
- Theorem: under the stated assumptions, pre- and post-activation features in every layer remain linearly independent at all training times.
- Corollary: if SGD converges, all error signals vanish, so the converged point is a global minimum.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use μP instead of SP, NTP, or IP | 3-hidden-layer MLP · CIFAR-10 | feature learning and feature richness | Helped |
| use SP instead of μP | 3-hidden-layer MLP · CIFAR-10 | feature change | Hurt |
| use NTP instead of μP | 3-hidden-layer MLP · CIFAR-10 | feature change | Hurt |
| use IP instead of μP | 3-hidden-layer MLP · CIFAR-10 | feature diversity | Hurt |
| use Tanh activation (ablation) | μP 3-hidden-layer MLP · CIFAR-10 | feature diversity as width increases | Hurt |
| use ReLU activation (ablation) | μP 3-hidden-layer MLP · CIFAR-10 | feature learning and representation richness | Helped |
| use μP instead of IP | 3-hidden-layer MLP · CIFAR-10, Layer 3 post-activation features | eigenvalue spectrum | Helped |
Surprising findings
- Standard and NTK parametrizations, despite keeping features rich in the reported metric, fail to learn features in the infinite-width limit.
- ReLU, which violates the smoothness assumptions, still shows feature learning empirically under μP without theoretical guarantee.
Limitations
- Analysis restricted to MLPs with full-batch or SGD and smooth activations; transformers and attention not covered.
- Assumes non-degenerate geometric conditions on training inputs (inner product distinctness), and ReLU not covered by theory.
- Experiments are small (10 samples, binary CIFAR-10 subset, 1000 steps), limited empirical validation.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: feature change (relative L2) minimum eigenvalue of feature Gram matrix eigenvalue spectrum percentiles
Baselines: standard parametrization neural tangent parametrization integrable parametrization (mean field)
Builds on: tensor programs maximal update parametrization neural tangent kernel mean field analysis
Closest papers by meaning
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.89)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.89)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.89)
- The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.88)
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.88)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.