Research map / Learning theory and algorithms / Kernels and regression theory

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel

ICML 2025 · Learning theory and algorithms › Kernels and regression theory · approach: Neural tangent kernel analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Random-matrix analysis of conjugate kernel (CK) and neural tangent kernel (NTK) of random-weight deep fully-connected networks on Gaussian-mixture inputs, showing isolated eigenvalues outside the limiting bulk whose eigenspaces carry class (group) information.

Problem

Understanding how isolated (spiked) eigenvalues and their eigenvectors of CK and NTK matrices arise and whether they encode class-group structure of the input data as features propagate through randomly weighted deep networks in the high-dimensional regime.

Approach

Builds on deterministic equivalents of CK/NTK (Gu et al., 2024) to reduce kernels to a low-rank deformation of XᵀX plus identity; applies finite-rank deformation random matrix theory to locate isolated eigenvalues (depending on activation functions and inter-class covariance differences) and derive the entrywise limit of the eigenspace projection onto class indicators.

Key results

• Isolated eigenvalues of CK and NTK appear outside the limiting spectral support when activations (e.g., ReLU) and inter-class covariance differences allow, with asymptotic positions given by alpha_{l,1} rho + alpha_{l,0} (CK) and beta_{l,1} rho + beta_{l,0} (NTK).
• Entrywise limit of the eigenspace projection onto class-indicator span is characterized (Theorem 3.4, Corollary 3.5), showing eigenvectors of isolated eigenvalues are informative for clustering.
• Simulations: no isolated eigenvalue for all-Poly activations, but one emerges with all-ReLU activations; MNIST digits 1 and 7 CK shows four isolated eigenvalues, with the top eigenvector informative.

Measured effects reported in the paper

Surprising findings

• With all-Poly activations no isolated eigenvalues appear, whereas ReLU yields one, showing activation choice alone determines whether spikes occur.

Limitations

• Results rely on Gaussian mixture model assumption for inputs; heavy-tailed or general distributions not covered.
• Analysis restricted to fully-connected architectures with random (untrained) weights and expected kernels in the infinite-width limit.

Experimental setup

Datasets: MNIST

Benchmarks: MNIST

Metrics: eigenvalue location eigenspace projection entries eigenvector informativeness

Baselines: Wang et al. (2024)

Builds on: Gu et al. (2024) deterministic equivalents for CK and NTK Benaych-Georges & Couillet (2016) Jacot et al. (2018) NTK Wang et al. (2024)

Closest papers by meaning

• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks (NeurIPS 2020 · similarity 0.93)
• Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures (ICML 2024 · similarity 0.88)
• On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels (NeurIPS 2022 · similarity 0.87)
• Equivariant Neural Tangent Kernels (ICML 2025 · similarity 0.86)
• Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability (NeurIPS 2025 · similarity 0.85)
• Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More (NeurIPS 2025 · similarity 0.85)