Research map / Label-efficient and robust learning / Recognition and long-tail
The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that infinite-width NTK Gram matrices, computed without training, can predict architecture quality, pseudo-label quality, class bias structure, and guide label refurbishment.
Problem
Expensive overparameterized model training requires iterative training for tasks like architecture selection, pseudo-label verification, bias identification, and label refurbishment; the paper asks whether these can be done before any model is trained.
Approach
Uses the Gram matrix of the infinite-width NTK (computed analytically via neural-tangents) as a single training-free signal: low-magnitude eigenvectors (KPCA) for architecture selection, inverse-Gram block-diagonalization error for pseudo-label verification, off-diagonal magnitudes for bias, and gradient-based discrete relabeling for label refurbishment.
Key results
- Training-free Gram-matrix computation is roughly 100x faster than training the matched finite-width model for small architectures and ~400x for larger CNNs (single A100).
- Architecture ranking predicted by Θ∞-KPCA achieves RBO 0.829 against brute-force training ranking on CIFAR-10 across 7 architectures.
- Infinite-width block-diagonalization error L(Z) ranks labeling schemes in the same order as actual training loss across four datasets; label refurbishment recovers 85.66% of 30% noise on Digit-MNIST vs BARE's 95.73% at the same noise level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the infinite-width block-diagonalization error to rank labeling schemes | Deep CNN training · Digit MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100 | training-loss ranking | Helped |
| Assign every training instance to one class instead of using clean labels | Deep CNN training · Digit MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100 | infinite-width block-diagonalization error: 6702.7 -> 1.3; 8558.3 -> 1.8; 1109.8 -> 0.2; 18456.4 -> 0.2 | Helped |
| Relabel noisy training data with the infinite-width NTK label gradient | Infinite-width NTK label refurbishment · Digit MNIST, 10% noise | mislabeled instances correctly refurbished: 83.50% vs 87.50% | Mixed |
| Relabel noisy training data with the infinite-width NTK label gradient | Infinite-width NTK label refurbishment · Digit MNIST, 20%, 30%, and 70% noise | mislabeled instances correctly refurbished: 85.00% vs 93.65%; 85.66% vs 95.73%; 25.21% vs 79.61% | Hurt |
| Relabel noisy training data with the infinite-width NTK label gradient | Infinite-width NTK label refurbishment · Fashion MNIST and CIFAR-10, 10%, 20%, 30%, and 70% noise | mislabeled instances correctly refurbished | Mixed |
| Predict architecture rankings with infinite-width NTK-KPCA | CIFAR-10 architecture selection · CIFAR-10 | architecture ranking: 0.829 RBO | Helped |
| Use the Gram-Label matrix to predict class rankings | Deep CNN bias identification · Digit MNIST and Fashion MNIST | Ranked Biased Overlap: 0.962; 0.904; 0.963; 0.915 | Helped |
| Use the Gram-Label matrix to predict class rankings | Deep CNN bias identification · CIFAR-10 and CIFAR-100 | Ranked Biased Overlap: 0.734; 0.916; 0.557; 0.688 | Mixed |
| Use an infinite-width architecture selected by NTK-KPCA rather than the 2-layer NTK | Complexity GAP label-noise detection · Digit MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100 | noise found after checking 20% of data: 93.70% -> 95.50%; 87.23% -> 91.25%; 54.05% -> 59.80%; 53.20% -> 59.35% | Helped |
| Use an infinite-width architecture selected by NTK-KPCA rather than the 2-layer NTK | Complexity GAP label-noise detection · Digit MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100 | AUC: .9955 -> .9978; .9823 -> .9899; .8397 -> .8765; .8387 -> .8760 | Helped |
Surprising findings
- Label refurbishment without training matches or underperforms BARE only partially; on CIFAR-10 at 70% noise only 11.5% of noise is correctly relabeled.
- A single-class labeling scheme is orders of magnitude easier to learn, as revealed by the NTK block-diagonalization error, because the model only optimizes bias terms.
Limitations
- Relabeling performance on CIFAR-10 and CIFAR-100 at high noise is low compared to some training-based methods, and BARE is competitive on Digit-MNIST.
- Results are on small image benchmarks with a limited set of architectures; the relation between infinite-width and finite-width behavior is only empirically supported.
- Requires unit-norm inputs to ensure positive definiteness of the kernel.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10 CIFAR-100 Shapes (synthetic) Corners (synthetic)
Benchmarks: MNIST Fashion-MNIST CIFAR-10 CIFAR-100
Metrics: RBO AUC percentage of noise found inter/intra-class distance ratio percentage of mislabeled data correctly relabeled
Baselines: BARE Complexity-Gap Exhaustive training ranking NAS methods
Builds on: neural tangent kernel Complexity-Gap (Nohyun et al. 2022) BARE (Patel and Sastry 2023) neural-tangents library
Closest papers by meaning
- Neural Tangent Kernels Under Stochastic Data Augmentation (AAAI 2026 · similarity 0.91)
- Fast Finite Width Neural Tangent Kernel (ICML 2022 · similarity 0.90)
- NTK-SAP: Improving neural network pruning by aligning training dynamics (ICLR 2023 · similarity 0.90)
- Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks (ICLR 2020 · similarity 0.90)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.89)
- Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.