Research map / Label-efficient and robust learning / Recognition and long-tail
A CLIP-Powered Framework for Robust and Generalizable Data Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a CLIP-based multimodal data selection framework that scores samples by image-text semantic alignment and local diversity, then selects subsets via SGD-based multi-objective optimization with a straight-through estimator to hit target ratios.
Problem
Static data selection for efficient training is typically based on image-only signals, which are unreliable under label noise and corruption, and selection by top/bottom scores ignores group effects among selected samples.
Approach
Adapts frozen CLIP image and text features with lightweight MLP adapters trained by InfoNCE, scores samples with a semantic alignment score (image-text cosine) and a KNN-based diversity score, and optimizes a relaxed binary selection vector with sigmoid and STE to meet the expected selection ratio.
Key results
- On CIFAR-100 with 20% symmetric label noise, the method reaches 46.05% accuracy vs. the best baseline Herding at 42.29%, while selecting only 0.24% noisy samples.
- On Tiny-ImageNet with VGG-16 at 70% selection ratio, the method reaches 53.40% accuracy vs. 50.79% for GraNd.
- On CIFAR-10 with ViT-small at 90% selection ratio, the method reaches 84.26% vs. 82.92% for MoSo and matches the full-data 84.00% accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP multimodal data selection | CIFAR-10, ResNet-50 at 20% selection · CIFAR-10 | top-1 accuracy: 84.60 -> 85.70 | Helped |
| Use CLIP multimodal data selection | CIFAR-10, ViT-small at 90% selection · CIFAR-10 | accuracy: 82.92 -> 84.26 | Helped |
| Use CLIP multimodal data selection | Tiny-ImageNet VGG-16 at 90% selection · Tiny-ImageNet | accuracy: 54.73 -> 56.34 | Helped |
| Use CLIP multimodal data selection | Tiny-ImageNet DenseNet-121 at 90% selection · Tiny-ImageNet | accuracy: 61.44 -> 61.59 | Helped |
| Use CLIP multimodal data selection | noisy-label CIFAR-100 at 20% selection · CIFAR-100 label noise | accuracy: 42.29 -> 46.05 | Helped |
| Use CLIP multimodal data selection | noisy-label Tiny-ImageNet at 30% selection · Tiny-ImageNet label noise | accuracy: 27.80 -> 33.13 | Helped |
| Add dataset adapters (ablation) | full CLIP selection method · CIFAR-100 | accuracy: 78.20 -> 78.98 | Helped |
| Add dataset adapters (ablation) | full CLIP selection method · Tiny-ImageNet | accuracy: 46.68 -> 49.30 | Helped |
| Add semantic alignment loss L_sa (ablation) | full CLIP selection method · CIFAR-100 | accuracy: 78.42 -> 78.98 | Helped |
| Add semantic alignment loss L_sa (ablation) | full CLIP selection method · Tiny-ImageNet | accuracy: 46.79 -> 49.30 | Helped |
| Add sample diversity loss L_sd (ablation) | full CLIP selection method · Tiny-ImageNet | accuracy: 49.14 -> 49.30 | Helped |
| Add selection-ratio loss L_s (ablation) | full CLIP selection method · Tiny-ImageNet | accuracy: 46.01 -> 49.30 | Helped |
Surprising findings
- Selecting 90% of Tiny-ImageNet data with the proposed method gives accuracy comparable to or slightly higher than full data, suggesting the method also acts as data cleaning.
- Replacing text features with average image prototypes for noise filtering increases the selected noise ratio sharply (e.g., 16.39% vs 0.24% at 20% selection ratio).
Limitations
- Potential CLIP biases may propagate into the selected dataset.
- Assumes balanced modalities; performance under modality-imbalanced datasets is untested.
- Requires category text labels, so applicability to unlabeled or non-classification data is limited.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k ImageNet-Hard ImageNet-R ImageNet-A
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k ImageNet-Hard ImageNet-R ImageNet-A
Metrics: top-1 accuracy top-5 accuracy noise proportion in selected data Dunn Index selection cost (hours)
Baselines: Random MoSo Glister Herding Forgetting GraNd EL2N Self-sup. prototypes CG-Score Moderate-DS
Builds on: CLIP InfoNCE straight-through estimator Moderate-DS CCS
Closest papers by meaning
- DsDm: Model-Aware Dataset Selection with Datamodels (ICML 2024 · similarity 0.87)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.87)
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.87)
- CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning (NeurIPS 2024 · similarity 0.86)
- BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges (ICML 2024 · similarity 0.86)
- PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.