Research map / Label-efficient and robust learning / Recognition and long-tail
Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies foundation models (DINOv2, CLIP, SigLIP, EVA-CLIP) as information extractors for one-shot subset selection and proposes RAM-APL, which fuses CLIP and DINOv2 features via ranking means and pseudo-class accuracy for fine-grained datasets.
Problem
One-shot subset selection relies on information extractors pre-trained on the full target dataset, which is dataset-dependent; it is unclear whether foundation models can replace these extractors and which FM to use.
Approach
Use multiple foundation models without alignment: map each model's features into a distance-ranking space (RAM) and a pseudo-class correctness score (APL), then combine them with sampling-rate-dependent weights to score samples for fine-grained subset selection.
Key results
- On CUB-200-2011, RAM-APL shows an average improvement of +6.40% over Random across sampling rates, versus +2.78% for the next best method (GC).
- On Pet, RAM-APL achieves +3.74% average improvement over Random vs. +1.52% for GC; on Food-101, +4.44% vs. +3.04% for GC.
- FMs were the optimal information extractor in 9 of 12 setups on Pet and Pet-N, but FMs showed limited advantage on coarse-grained noisy CIFAR-10N (4 of 12 setups).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAM-APL with CLIP and DINOv2 rather than Random selection | ResNet-18 subset training · Oxford-IIIT Pet | mean accuracy improvement over Random across sampling rates: +3.74 | Helped |
| Use RAM-APL with CLIP and DINOv2 rather than Random selection | ResNet-18 subset training · Food-101 | mean accuracy improvement over Random across sampling rates: +4.44 | Helped |
| Use RAM-APL with CLIP and DINOv2 rather than Random selection | ResNet-50 subset training · CUB-200-2011 | mean accuracy improvement over Random across sampling rates: +6.40 | Helped |
| Add APL to RAM (ablation) | RAM subset selection with CLIP+DINOv2 · Pet, 1% and 50% sampling | accuracy: 5.9±0.3 -> 6.5±0.4; 47.1±1.4 -> 47.5±1.9 | Helped |
| Add APL to RAM (ablation) | RAM subset selection with CLIP+DINOv2 · Pet, 70% sampling | accuracy: 56.5±2.7 -> 58.7±2.2 | Helped |
| Replace ranking-based feature fusion with feature concatenation (ablation) | RAM-APL with CLIP+DINOv2 · Pet, sampling rates 1%, 10%, 30%, 50%, and 70% | accuracy: 5.9±0.4 vs 6.5±0.4; 16.3±0.4 vs 15.2±1.2; 31.7±1.3 vs 32.4±2.9; 47.7±3.0 vs 47.5±1.9; 57.8±1.2 vs 58.7±2.2 | Mixed |
| Use CLIP+DINOv2 together instead of either model individually (ablation) | RAM-APL subset selection · Pet | accuracy: Single and multiple model scores vary by sampling rate; e.g., 47.7±1.1 (DINOv2), 43.6±1.9 (CLIP), 47.5±1.9 (DINOv2+CLIP) | Mixed |
| Use RAM-APL subsets with CLIP+DINOv2 instead of baseline-selected subsets | MobileNet-V3 target model · Pet, 10%, 30%, and 50% sampling | accuracy: 10.9±1.1 -> 13.6±0.3; 42.1±3.6 -> 45.7±0.9; 61.6±1.9 -> 62.3±1.4 | Helped |
| Use a foundation model rather than a traditional IE | One-shot subset selection · Fine-grained datasets, including clean and noisy Pet | prediction accuracy | Helped |
| Use a foundation model rather than a traditional IE | One-shot subset selection · Noisy coarse-grained datasets | prediction accuracy | No clear effect |
| Choose a single FM based only on its downstream-task accuracy | FM-based subset selection · Fine-grained datasets across selection methods and sampling rates | subset selection prediction accuracy | No clear effect |
| Use RAM-APL rather than MIN with DINOv2 (ablation) | Pet subset selection · Pet, 1% and 50% sampling | accuracy: 6.2±0.1 -> 6.5±0.4; 46.8±2.0 -> 47.5±1.9 | Helped |
Surprising findings
- Superior downstream zero-shot/target accuracy of an FM (e.g., EVA-CLIP on Pet) does not guarantee better subset selection performance.
- Simple feature concatenation of multiple FMs performs comparably to the proposed fusion at several sampling rates, with the proposed method winning mainly at 1%, 30%, and 70%.
Limitations
- Evaluation limited to three fine-grained image datasets and ResNet-type targets with a small number of seeds for Food-101 and CUB.
- Gains over baselines at some sampling rates are within reported standard deviations (e.g., Pet at 10%, CUB at 30%).
- Hyperparameters alpha and beta selected on the evaluation setting; sensitivity shown but no principled selection rule.
Experimental setup
Datasets: Oxford-IIIT Pet Food-101 CUB-200-2011 CIFAR-10 CIFAR-10N CIFAR-10I CIFAR-100 TinyImageNet
Benchmarks: Oxford-IIIT Pet Food-101 CUB-200-2011
Metrics: test accuracy of target model average improvement over Random
Baselines: Random Herding K-Center Greedy Contextual Diversity Margin Forgetting GraNd Cal Glister Graph Cut Moderate_DS MIN
Builds on: DeepCore K-center greedy Graph Cut Moderate_DS Herding
Closest papers by meaning
- Balancing Feature Similarity and Label Variability for Optimal Size-Aware One-shot Subset Selection (ICML 2024 · similarity 0.88)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.86)
- Curriculum Coarse-to-Fine Selection for High-IPC Dataset Distillation (CVPR 2025 · similarity 0.85)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.84)
- BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges (ICML 2024 · similarity 0.84)
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.