Research map / Learning theory and algorithms / Kernels and regression theory
PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Incremental Data Selection (IDS), where a stream of labeled examples is selected under a fixed budget while the model trains, and proposes PEAKS, which scores examples by prediction error times kernel (feature) similarity to class prototypes.
Problem
Selecting a fixed-size training subset from a stream of labeled examples without access to the full data source, while the model is trained incrementally on the selected examples.
Approach
Derives from a first-order NTK/last-layer analysis that an example's utility is proportional to its prediction error times its feature similarity to the class mean embedding; PEAKS approximates the class mean with output-layer weights so no validation set is needed, and uses percentile-based dynamic thresholds on a score cache.
Key results
- On WebVision, PEAKS with 50k and 100k samples matches random selection using 200k and 400k samples (about 4x data efficiency), per the scaling experiments.
- With validation set and ViT-B/16 (DINO), PEAKS-V reaches 59.9% on WebVision at x4 budget vs 50.0% for random, and 73.1% vs 69.7% on Food101 at x4.
- Without validation set, PEAKS reaches 59.0% on WebVision at x4 vs 50.1% random and 54.8% Moderate Emb.; on CIFAR100 wrong-low-confidence outperforms PEAKS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PEAKS instead of the strongest listed baseline at each data budget | ViT-B/16 (DINO, ImageNet-1k) · Food101, Food101-N, and WebVision; without validation set | test accuracy | Helped |
| Use PEAKS instead of the strongest listed baseline | ViT-B/16 (DINO, ImageNet-1k) · CIFAR100; without validation set | test accuracy: 59.0 ($\pm 6.5$) vs 63.3 ($\pm 5.9$); 72.3 ($\pm 3.2$) vs 77.0 ($\pm 2.8$); 82.7 ($\pm 0.9$) vs 83.9 ($\pm 1.0$) | Mixed |
| Use PEAKS-V with validation-set class means rather than distance-based embedding baselines | ViT-B/16 (DINO, ImageNet-1k) · CIFAR100, Food101, Food101-N, and WebVision; with validation set | test accuracy | Helped |
| Use PEAKS with output-layer weight prototypes instead of PEAKS-V with validation-set class means (ablation) | PEAKS · Fine-tuning datasets; limited samples per class (e.g., ×1) | test accuracy | Hurt |
| Set selection rate to p = 10% rather than the less-strict rates evaluated (ablation) | PEAKS and other selection methods · Food101 and Food101-N | test accuracy | Hurt |
| Increase selection rate; PEAKS peaks at p = 30% (ablation) | PEAKS · Food101 and Food101-N | test accuracy | Helped |
| Increase refresh period τ across the evaluated range (ablation) | PEAKS and other selection methods · Food101 and Food101-N | test accuracy | No clear effect |
Surprising findings
- On the clean CIFAR100 dataset, the simple wrong-and-low-confidence baseline outperforms PEAKS, and random selection is a surprisingly strong baseline with small budgets when training from scratch.
- Hard-embedding selection performs very poorly on noisy datasets (e.g., 16.0% on WebVision at x4 with validation set), below random.
Limitations
- Derivation assumes last-layer training and kernel regime, while experiments train the full architecture.
- Training from scratch gives only modest gains over random; performance degrades without a validation set when samples per class are very limited.
- Gap between PEAKS and upper bounds on noisy, complex datasets (Food101-N, WebVision) indicates room for improvement.
Experimental setup
Datasets: CIFAR100 Food101 Food101-N WebVision CIFAR10
Benchmarks: CIFAR100 Food101 Food101-N WebVision
Metrics: test accuracy Jaccard similarity
Baselines: Random Easy Embedding Hard Embedding Moderate Embedding EL2N GraNd Uncertainty Wrong Low Confidence
Builds on: EL2N GraNd neural tangent kernel DINO
Closest papers by meaning
- Diversified Batch Selection for Training Acceleration (ICML 2024 · similarity 0.86)
- DsDm: Model-Aware Dataset Selection with Datamodels (ICML 2024 · similarity 0.86)
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.85)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.85)
- Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning (ICLR 2024 · similarity 0.85)
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.