Research map / Learning theory and algorithms / Kernels and regression theory

Double-Descent Curves in Neural Networks: A New Perspective Using Gaussian Processes

AAAI 2024 · Learning theory and algorithms › Kernels and regression theory · approach: Double descent and overfitting · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses random matrix theory to derive the spectrum of a width-dependent empirical NNGP kernel and shows that double descent in GP and kernel regression arises from the discrepancy between this spectrum and the width-independent NNGP kernel spectrum.

Problem

Explaining double-descent in the generalisation error of neural networks as a function of width relative to training set size, which prior random-matrix analyses handled only with width taken to infinity or with random features models.

Approach

Defines a finite-width random kernel (empirical NNGP kernel) whose spectrum is the Marchenko-Pastur map of the infinite-width NNGP kernel spectrum; double descent is then a perturbation of the NNGP spectrum that vanishes as gamma tends to zero, with divergence at the interpolation threshold.

Key results

• Simulated empirical spectral distributions of the kernel random matrix match the Marchenko-Pastur-map theoretical limit for a two-layer NNGP on synthetic data (N=300, n=200, d=400) and MNIST 0/1 (N=600, n=300, d=784).
• Theory predicts double descent in NNGP kernel regression generalisation error as a function of gamma = n/N, with finite overparameterized asymptote and divergence at the interpolation threshold gamma=1; simulations show reasonable agreement though not exact.
• Generalisation error coefficients were estimated via the spectral universality assumption, so numerical predictions are approximate, while the qualitative double-descent prediction holds.

Measured effects reported in the paper

Limitations

• Generalisation error coefficients rely on the spectral universality assumption, which is not proven in general.
• Numerical predictions for generalisation error are not perfectly accurate on MNIST, attributed to a small 300-example subset and SUA mismatch.
• Analysis covers only the width of the last layer going to infinity and fully connected zero-bias teacher-student setting with the last layer as the finite width.

Experimental setup

Datasets: MNIST (digits 0 and 1 subset)

Benchmarks: MNIST (digits 0 and 1 subset) synthetic isotropic Gaussian data

Metrics: empirical spectral distribution agreement generalisation error of NNGP kernel regression

Baselines: Fan and Wang 2020 Louart, Liao, and Couillet 2018 Canatar et al. 2021 Jacot et al. 2020

Builds on: neural network Gaussian process Mei and Montanari 2019 empirical covariance Marchenko-Pastur law Fan and Wang 2020

Closest papers by meaning

• Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.86)
• Generalization error of spectral algorithms (ICLR 2024 · similarity 0.86)
• Sharp Generalization for Nonparametric Regression by Over-Parameterized Neural Networks: A Distribution-Free Analysis in Spherical Covariate (ICML 2025 · similarity 0.86)
• Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space (ICLR 2024 · similarity 0.86)
• Free Decompression with Algebraic Spectral Curves (NeurIPS 2026 · similarity 0.85)
• Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.85)