Research map / Learning theory and algorithms / Kernels and regression theory
How Spurious Features are Memorized: Precise Analysis for Random and NTK Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a precise theory of how spurious (label-independent) features are memorized in over-parameterized generalized linear regression, showing memorization is governed by stability times a feature-alignment constant that is proven to concentrate for random features and NTK models.
Problem
Spurious features that are uncorrelated with the label are memorized by overparameterized models, but no rigorous theory quantifies this memorization or the role of architecture and activation.
Approach
Decomposes spurious-feature memorization into model stability with respect to a training sample and a novel feature alignment between the spurious sample and the training sample; shows the alignment concentrates to a constant for random features and NTK regression, with a closed form for NTK that depends on the activation derivative's Hermite coefficients.
Key results
- For random features, feature alignment concentrates to a constant gamma_RF bounded below by sum_{l>=2} mu_l^2 alpha^l / sum_{l>=1} mu_l^2, so spurious memorization is positive whenever alpha > 0.
- For NTK, the alignment concentrates to a closed-form gamma_NTK = alpha * sum_l mu'_l^2 alpha^l / sum_l mu'_l^2 in (0,1).
- Experiments on MNIST and CIFAR-10 show spurious accuracy decreases as training size N grows and is lower for activations with dominant high-order Hermite coefficients.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the number of training samples | Random features model · Synthetic Gaussian binary classification | test accuracy | Helped |
| increase the number of training samples | Random features model · Synthetic Gaussian binary classification | spurious accuracy | Hurt |
| increase the spurious-feature fraction α | Random features model · Synthetic Gaussian binary classification | spurious accuracy | Helped |
| use an activation function with dominant low-order Hermite coefficients | Random features model · Synthetic Gaussian binary classification | spurious accuracy | Helped |
| vary α or the activation function | Random features model · Synthetic Gaussian binary classification | test accuracy | No clear effect |
| increase the number of training samples | NTK model · MNIST and CIFAR-10 | test accuracy | Helped |
| increase the number of training samples | NTK model · MNIST and CIFAR-10 | spurious accuracy | Hurt |
| use an activation derivative with higher-order Hermite coefficients | NTK model · MNIST and CIFAR-10 | spurious accuracy | Hurt |
Surprising findings
- Spurious features are memorized even when they are uncorrelated with the label, as long as generalization error is nonvanishing.
Limitations
- Analysis restricted to generalized linear regression (RF and NTK) with a quadratic loss and gradient descent in the lazy/interpolating regime.
- Assumes spurious feature independent of label; correlated spurious features not covered.
Experimental setup
Datasets: MNIST CIFAR-10 synthetic Gaussian data
Benchmarks: MNIST CIFAR-10
Metrics: test accuracy spurious accuracy
Builds on: neural tangent kernel random features leave-one-out stability
Closest papers by meaning
- The Pitfalls of Memorization: When Memorization Hurts Generalization (ICLR 2025 · similarity 0.89)
- Complexity Matters: Feature Learning in the Presence of Spurious Correlations (ICML 2024 · similarity 0.88)
- Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples (CVPR 2025 · similarity 0.87)
- Towards Memorization Estimation: Fast, Formal and Free (ICML 2025 · similarity 0.87)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.86)
- On the Over-Memorization During Natural, Robust and Catastrophic Overfitting (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.