Research map / Learning theory and algorithms / Clustering
Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a k-means-clustering-plus-sensitivity-sampling data selection method that provably preserves average model loss under a Hölder continuity assumption, with experiments on T5 translation fine-tuning, image classification, and linear regression.
Problem
Selecting a small representative subset of data to train or fine-tune a model when evaluating per-example losses on the full dataset is too expensive, especially for foundation models; prior k-center coreset guarantees are weak and limited to classification.
Approach
Cluster the data embeddings with k-means, query the loss only at cluster centers, extrapolate losses to all points using a Hölder continuity bound, and sample points by sensitivity (cluster distance plus center loss), yielding an unbiased weighted loss estimate with error bounded by the k-means cost.
Key results
- On WMT EnDe fine-tuning of T5-Small with ~1% of data, the method improves evaluation accuracy from a 0.59 random baseline to about 0.6, covering about 9% of the headroom to the full-data 0.7 accuracy.
- For classification, the method outperforms baselines by several percentage points, with more than 4% accuracy gain on Fashion MNIST over other methods.
- On linear regression (UCI gas sensor), the clustering-based sampler performs almost equally well as leverage score sampling while being much faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sensitivity sampling instead of uniform random sampling | T5-Small fine-tuning · WMT T2T EnDe | accuracy: 0.59 -> 0.6 | Helped |
| Use Sentence-T5 rather than BERT embeddings for sampling (ablation) | Sensitivity sampling for T5-Small fine-tuning · WMT T2T EnDe | accuracy | No clear effect |
| Raise the Hölder constant to 500 (ablation) | Sensitivity sampling for T5-Small fine-tuning · WMT T2T EnDe | accuracy | Hurt |
| Use loss-based rather than gradient-based sampling (ablation) | Sensitivity sampling for image classification · MNIST, Fashion MNIST, and CIFAR10 | runtime | Helped |
Surprising findings
- A simple BERT-embedding clustering is predictive of T5 model losses, with losses of cluster-similar points closer than random points.
- Raising the Hölder constant to 500 degrades the translation results toward diversity-sampling performance.
Limitations
- Guarantees depend on the Hölder/well-behaved assumption and clusterability of embeddings, which must be validated empirically.
- Hölder constant must be estimated or tuned; performance sensitive to its value.
- Gains on the translation task are modest (about 9% of headroom to full-data training).
Experimental setup
Datasets: WMT T2T EnDe MNIST Fashion MNIST CIFAR-10 UCI gas sensor
Benchmarks: WMT T2T EnDe MNIST Fashion MNIST CIFAR-10 UCI gas sensor
Metrics: accuracy BLEU R^2 score runtime
Baselines: Uniform sampling Diversity sampling (k-means based) Margin sampling Entropy sampling Leverage score sampling Random-deduped
Builds on: coresets for clustering k-center coreset selection (Sener and Savarese) k-means++ DISTIL benchmark
Closest papers by meaning
- Towards a statistical theory of data selection under weak supervision (ICLR 2024 · similarity 0.88)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.87)
- Diversified Batch Selection for Training Acceleration (ICML 2024 · similarity 0.86)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.86)
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.86)
- Structural-Entropy-Based Sample Selection for Efficient and Effective Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.