Research map / Label-efficient and robust learning / Self-supervised and contrastive
SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SELECT, a benchmark of data curation strategies for image classification, and ImageNet++, five ImageNet-scale training shifts (OpenImages, LAION img2img/txt2img, Stable Diffusion img2img/txt2img) evaluated by training over 130 models against the expert-curated ImageNet-1K.
Problem
Data curation strategies for image classification are rarely compared systematically at scale; it is unclear how reduced-cost curation (embedding search, synthetic generation, crowdsourcing) compares with expert-labeled ImageNet-1K.
Approach
A benchmark that fixes the model, training recipe, and label set, varies only the curated training data, and measures utility (base accuracy, OOD robustness, VTAB fine-tuning, DINO kNN) plus analytic metrics (imbalance, quality, correlations) across curation strategies, instantiated via the ImageNet++ shifts.
Key results
- Expert-curated ImageNet-1K remains the best baseline: IN1000 reaches 77.9% ImageNet-Val accuracy versus 59.6% for LA1000 (img2img), the best reduced-cost shift.
- OI1000 scores 44.1 on Avg. VTAB, above IN1000's 41.2, showing that base accuracy does not track fine-tuning utility.
- Under SSL kNN evaluation at SPC=5, OI1000 (24.61) is closer to IN1000 (31.77) than it is at higher SPC, and SD1000 (txt2img) performs strongly at low SPC (30.40).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace expert-curated IN1000 with reduced-cost curated data | ResNet-50 image classifier · IN1000-Val | accuracy: 77.9 $\pm$ 0.4 vs 59.6 $\pm$ 0.4 (LA1000 img2img); 42.3 $\pm$ 0.4 (OI1000) | Hurt |
| Use LA1000 img2img instead of LA1000 txt2img | ResNet-50 image classifier · IN1000-Val | accuracy: 55.7 $\pm$ 0.4 -> 59.6 $\pm$ 0.4 | Helped |
| Use LA1000 img2img instead of LA1000 txt2img | ResNet-50 image classifier · Avg. Nat. Rob. | accuracy: 26.0 $\pm$ 0.4 -> 23.2 $\pm$ 0.5 | Hurt |
| Use LA1000 img2img instead of OI1000 | ResNet-50 image classifier · Avg. VTAB | accuracy: 44.1 $\pm$ 0.8 -> 41.1 $\pm$ 0.8 | Hurt |
| Use class-balanced weighted loss for the IN1000 + OI1000 blend | ResNet-50 image classifier · IN1000 Avg. Rob. | accuracy: 18.3\% to 35.7\% | Helped |
| Blend OI1000 data with IN1000 data to rebalance classes | ResNet-50 image classifier · IN1000-Val | accuracy: 31.3\pm 0.41\% -> 77.1\pm0.37\% (IN1000 + OI1000) | Mixed |
| Increase OI1000 share and introduce long-tailed classes | ResNet-50 image classifier, 1000 classes · IN1000-Val | accuracy: 69.1 $\pm$ 0.08\% -> 30.0 $\pm$ 0.08\% | Hurt |
| Increase samples per class in kNN guidance from SPC=5 to SPC=1000 | DINO ResNet-50 · IN1000-Val | accuracy: 31.77 -> 66.72 (IN1000); 27.39 -> 58.91 (LA1000 img2img) | Helped |
| Increase samples per class in kNN guidance from SPC=5 to SPC=1000 | DINO ResNet-50 · IN1000-Val | accuracy: 30.40 -> 40.93 (SD1000 txt2img) | Mixed |
| Use SD1000 txt2img instead of SD1000 img2img | ResNet-50 image classifier · Avg. SSL | accuracy: 28.8 $\pm$ 0.2 -> 35.8 $\pm$ 0.2 | Helped |
| Use SD1000 txt2img instead of SD1000 img2img | ResNet-50 image classifier · IN1000-Val | accuracy: 25.2 $\pm$ 0.4 -> 23.7 $\pm$ 0.4 | Hurt |
| Add OI1000 data to IN1000 and apply weighted loss | ResNet-50 image classifier · IN1000 Avg. Rob. | accuracy: 33.7\% -> 35.7\% | Helped |
Surprising findings
- Human-labeled OI1000 is outperformed by the cheaper LA1000 (img2img) on most metrics, which the authors attribute to label imbalance.
- Inception Score favors synthetic SD1000 images over real images, contradicting its use as an image quality proxy.
Limitations
- Covers only six curation methods, one task (image classification), and one label set (ImageNet).
- Cost modeling is coarse, with only low and high bins.
- Image quality metrics are weak predictors of downstream utility.
Experimental setup
Datasets: ImageNet-1K OpenImages LAION Stable Diffusion generated images
Benchmarks: ImageNet-Val ImageNet-Sketch ObjectNet ImageNet-V2 ImageNet-R ImageNet-A ImageNet-C Stylized-ImageNet VTAB-1k
Metrics: top-1 accuracy average natural robustness average synthetic robustness kNN accuracy LT@500 LS@5pct CLIPScore CLIP-IQA Inception Score CMMD
Baselines: ImageNet-1K (expert curated) OI1000 (OpenImages crowdsourced) LA1000 (img2img, LAION embedding search) LA1000 (txt2img, LAION embedding search) SD1000 (img2img, Stable Diffusion) SD1000 (txt2img, Stable Diffusion)
Builds on: DINO CLIP VTAB-1k DataComp
Closest papers by meaning
- Data Filtering Networks (ICLR 2024 · similarity 0.87)
- Doomed to Re-Annotate, Forever: The ImageNet Story (NeurIPS 2026 · similarity 0.87)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.86)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.86)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.86)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.