Research map / Optimization / Implicit bias and generalization

Asymmetric Scaling Laws from Sparse Features

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Gradient flow and mean-field dynamics · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a sparse random-feature linear model with heavy-tailed Bernoulli input activations to show that sparsity yields two distinct scaling exponents (model-limited vs data-limited), a double-descent peak, a compute-optimal frontier favoring data, and a scaling law for GD instability.

Problem

Explaining why neural scaling laws can have asymmetric exponents for model size and data size, and how sparse feature activation affects population loss, compute-optimal allocation, and gradient descent behavior.

Approach

Sparse coordinate activations mean rare coordinates are never observed in the training set, creating a data-limited bottleneck with exponent (alpha1+alpha2+1)/(alpha1+1) distinct from the model-limited exponent alpha1+alpha2+1; this asymmetry arises purely from input sparsity in a sketched linear regression model.

Key results

• Underparameterized exponent alpha_N = alpha1+alpha2+1 and overparameterized exponent alpha_D = (alpha1+alpha2+1)/(alpha1+1), with alpha_D < alpha_N for alpha1>0.
• Compute-optimal exponent alpha_C = (alpha1+alpha2+1)/(alpha1+3), with optimal allocation shifting toward data as alpha1 increases.
• In a ReLU two-layer network with (alpha1, alpha2)=(1.0, 0.3), the sparse D-sweep fits a shallower exponent alpha_D ~ 1.2 versus ~1.5 for the other sweeps.

Surprising findings

• Compute-optimal allocation always lies in the underparameterized regime, so no overparameterized optimum exists under the assumed compute model.
• Near-square-root compute-optimal exponents can arise from only a small asymmetry between alpha_N and alpha_D.

Limitations

• Results rely on Gaussian-equivalence-free heuristics and a near-isometry assumption for the random embedding in the overparameterized regime.
• Nonlinear case is validated only empirically, without theory; label noise and ridge regularization are not analyzed.
• Compute model C = N D min(N,D) is an adopted proxy rather than measured cost.

Experimental setup

Datasets: synthetic sparse Bernoulli-power-law data

Benchmarks: synthetic sparse random-feature regression

Metrics: population test loss scaling exponents

Baselines: dense power-law random features

Builds on: Chinchilla compute-optimal scaling random feature models double descent

Closest papers by meaning

• A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.90)
• Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.90)
• Unified Scaling Laws for Compressed Representations (NeurIPS 2025 · similarity 0.89)
• Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.89)
• 4+3 Phases of Compute-Optimal Neural Scaling Laws (NeurIPS 2024 · similarity 0.89)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.88)