Research map / Learning theory and algorithms / Kernels and regression theory
The $\varphi$ Curve: The Shape of Generalization through the Lens of Norm-based Capacity Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives deterministic equivalents for the ℓ2 norm of random-feature (ridge) regression estimators and uses them to show the test risk vs. norm curve is U-shaped (no double descent) under norm-based capacity.
Problem
Classical model-size capacity fails to explain learning curves (double descent) of over-parameterized models; the paper asks how test risk relates to norm-based capacity and whether its shape is U-shaped or double-descent.
Approach
Uses deterministic equivalence from random matrix theory to derive asymptotic and non-asymptotic deterministic equivalents of the estimator's ℓ2 norm for random features ridge regression, then relates test risk to norm; under norm capacity the risk curve is U-shaped (increasing with norm in over-parameterized regime) rather than double-descent in model size.
Key results
- Theory predicts a phase transition from under- to over-parameterization; in the over-parameterized regime test risk increases almost linearly with the estimator norm for min-norm interpolation, the opposite of double descent in model size.
- Norm-based capacity of the second layer captures both bias and variance of the test risk, whereas effective dimension and smoother capture only variance.
- Scaling law under norm capacity takes the form R = Theta(n^{gamma_n} N^{gamma_N}) with gamma_n <= 0 and gamma_N that can be positive or negative, so the risk is not monotone in the norm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| measure capacity by the second-layer estimator's ℓ2 norm instead of model size | Random feature models · Under- and over-parameterized regimes | test risk versus capacity | Mixed |
| use effective dimension instead of the estimator norm to characterize generalization | Random feature models · Under- and over-parameterized regimes | test-risk characterization | Hurt |
| use the smoother instead of the estimator norm to characterize generalization | Random feature models · Under- and over-parameterized regimes | test-risk characterization | Hurt |
| increase the estimator norm in the over-parameterized regime | Random feature min-norm estimator · Over-parameterized regime | test risk | Hurt |
| increase model size in the over-parameterized regime | Random feature min-norm estimator · Over-parameterized regime | test risk | Helped |
| use sufficient over-parameterization rather than under-parameterization | Random feature models · Under- and over-parameterized regimes | test risk | Helped |
| increase regularization strength λ | Random feature ridge regression · Under- and over-parameterized regimes | estimator norm | Helped |
| increase the estimator norm by tuning regularization | Random feature ridge regression · Under- and over-parameterized regimes | test risk | Mixed |
| use a larger norm in low-complexity scaling-law regions | Random feature models · Scaling law with p = n^q | test risk | Helped |
| use a larger norm in high-complexity scaling-law regions | Random feature models · Scaling law with p = n^q | test risk | Hurt |
| measure capacity by the untrained first-layer weight norm instead of the trained second-layer norm | Random feature models · Test risk versus capacity | test-risk curve shape | Hurt |
Surprising findings
- In the over-parameterized regime, test risk increases with estimator norm, so the curve is not monotone in the way classical double descent is in model size.
- The first-layer weight Frobenius norm yields a reshaped double descent instead of a U-shape.
Limitations
- Analysis is limited to linear and random features models; deep networks are not covered.
- Theory does not fully reproduce the phi-shaped curve on real datasets because real data may violate the well-behaved data assumption.
- Results rely on asymptotic or technical non-asymptotic assumptions (sub-Gaussian/concentration, isotropic features for closed forms).
Experimental setup
Datasets: synthetic data real-world datasets (unspecified in text) FashionMNIST (appendix)
Metrics: test risk bias variance estimator norm
Baselines: double descent (model size) effective dimension smoother
Builds on: deterministic equivalence (random matrix theory) random features models double descent
Closest papers by meaning
- The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (ICML 2019 · similarity 0.86)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.86)
- The role of over-parametrization in generalization of neural networks (ICLR 2019 · similarity 0.86)
- Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods (ICLR 2025 · similarity 0.85)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.84)
- Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.