Research map / Label-efficient and robust learning / Recognition and long-tail
lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes labelled pseudo NTK (lpNTK), a label-aware sample-interaction kernel, and uses it to explain learning difficulty and forgetting and to prune redundant or head-cluster samples for better generalisation on MNIST and CIFAR-10.
Problem
How samples interact during neural network training and how data relationships (including labels) affect learning difficulty, forgetting, and generalisation; existing pNTK ignores label information.
Approach
Approximate sample-to-sample interaction via first-order Taylor expansion of SGD, observe that labels flip signs of the eNTK entries, and define lpNTK as a label-sign-weighted sum of eNTK entries, interpreted as a linear kernel on label-signed gradient features; classify sample pairs as interchangeable, unrelated, or contradictory.
Key results
- Removing lpNTK-redundant samples keeps test accuracy statistically unchanged (MNIST 99.30% vs 99.31% full; CIFAR10 93.17% vs 93.28%), whereas pNTK-based removal drops CIFAR10 to 90.93%.
- Pruning 10% of the largest FPC cluster after lpNTK de-redundancy (about 20% total removed) improves test accuracy: CIFAR10 93.55% vs 93.28% full, MNIST 99.37% vs 99.31%.
- lpNTK-based predictions of forgetting events reach F1 of 49.54% on MNIST and 57.76% on CIFAR-10, which the authors describe as significantly better than random guessing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove lpNTK-redundant samples and randomly prune 10% of the largest cluster | LeNet-5 · MNIST | test accuracy: 99.31(± 0.03)% -> 99.37(± 0.04)% | Helped |
| remove lpNTK-redundant samples and randomly prune 10% of the largest cluster | ResNet-18 · CIFAR-10 | test accuracy: 93.28(± 0.06)% -> 93.55(± 0.12)% | Helped |
| remove samples identified as redundant by lpNTK | LeNet-5 · MNIST | test accuracy: 99.31% -> 99.30% | No clear effect |
| remove samples identified as redundant by lpNTK | ResNet-18 · CIFAR-10 | test accuracy: 93.28% -> 93.17% | No clear effect |
| remove samples identified as redundant by pNTK | LeNet-5 · MNIST | test accuracy: 99.31% -> 99.27% | Hurt |
| remove samples identified as redundant by pNTK | ResNet-18 · CIFAR-10 | test accuracy: 93.28% -> 90.93% | Hurt |
Surprising findings
- Removing redundant samples identified by pNTK hurts CIFAR10 accuracy noticeably, while the lpNTK version does not.
- The authors found contradictory sample pairs to be rare in practice.
Limitations
- Number of FPC clusters and the pruning fraction (10% of head cluster) are heuristic hyperparameters with no theoretical guarantee.
- Evaluation is limited to small image benchmarks (MNIST, CIFAR-10) with LeNet-5 and ResNet-18.
- Taylor approximation accuracy depends on learning rate and model size; hyperparameters were not tuned, so absolute accuracies are lower than reported elsewhere.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: test accuracy Pearson correlation coefficient precision recall F1-score t-test p-value
Baselines: pNTK EL2N GraNd Forgot Score
Builds on: pseudo NTK empirical NTK forgetting events EL2N score
Closest papers by meaning
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · similarity 0.85)
- What can linearized neural networks actually say about generalization? (NeurIPS 2021 · similarity 0.85)
- Neural Networks as Kernel Learners: The Silent Alignment Effect (ICLR 2022 · similarity 0.85)
- Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (ICML 2020 · similarity 0.84)
- The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training (AAAI 2025 · similarity 0.84)
- How Graph Neural Networks Learn: Lessons from Training Dynamics (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.