Research map / Learning theory and algorithms / Kernels and regression theory

Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

NeurIPS 2026 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional asymptotic generalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives via a heuristic leave-one-out (cavity) argument fixed-point equations for feature overlaps and Bayes-optimal error in a one-hidden-layer teacher-student model with extensive width k=Θ(d), showing sequential feature-learning phase transitions and Bayes-optimal scaling laws unified by an effective width k_c.

Problem

Information-theoretic limits of learning a one-hidden-layer teacher with hierarchical (power-law) readout weights from noisy queries when teacher width scales linearly with input dimension, and how to size a student at a given data budget.

Approach

A leave-one-out cavity argument reduces multi-feature inference to independent scalar GLMs, giving closed fixed-point equations for per-feature overlaps and BO error for arbitrary readouts and activations with vanishing second Hermite coefficient. Features become learnable sequentially via SNR thresholds, yielding an effective width k_c with BO error Θ(k_c d/n); a student of width near k_c trained

Key results

• Bayes-optimal error scales as n^{1/(2β)-1} in the feature-learning regime and n^{-1} in the refinement regime, both unified by ε^BO=Θ(k_c d/n).
• Adam students near the effective width k_c approximately track the Bayes-optimal data-size exponents, with about 5% of the weakest features remaining unlearned under Adam.
• Predicted optimal model-size exponent 1-2β for SGD students below the critical width, checked empirically in the middle panel of the figure.

Surprising findings

• A reduced-width student can outperform the full-width student because non-learnable teacher features act as effective noise.
• Under Adam, roughly 5% of the weakest features remain non-learnable at large n, causing the refinement exponent to deteriorate.

Limitations

• Derivations are heuristic, relying on unproved Gaussian cavity-field and overlap concentration assumptions.
• Restricted to activations with vanishing second Hermite coefficient (mu_2=0), e.g. odd activations.
• Single hidden layer only; multi-layer extension left open.

Experimental setup

Datasets: synthetic Gaussian inputs with teacher-generated noisy labels

Metrics: generalization error (MSE) training error feature overlap

Baselines: Adam-trained full-width student optimized-width Adam student Bayes-optimal error

Builds on: replica method for dense-readout extensive-width networks approximate message passing quantization model of neural scaling

Closest papers by meaning

• On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.91)
• A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.90)
• The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.89)
• Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.89)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.89)
• The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.88)