Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLYT, which trains a scoring model to weight pretraining examples via downstream (ImageNet) gradient signals through a reference CLIP model, plus M-FLYT (learned linear mixing of existing scores) and Soft Cap Sampling for filtering DataComp medium.
Problem
Data filtering for CLIP pretraining: existing methods rely on heuristic proxies for example usefulness; the goal is to learn a data-driven measure of each pretraining example's usefulness for downstream performance under the DataComp filtering benchmark.
Approach
Learn a scoring model whose softmax weights define a weighted CLIP loss; a one-step virtual update of a reference CLIP model on upstream data is evaluated on downstream data, and the downstream gradient flows back to the scoring model. M-FLYT learns a linear mix of existing scores as features, and Soft Cap Sampling samples with replacement while subtracting a repetition penalty from scores.
Key results
- M-FLYT with SCS reaches 40.1% ImageNet zero-shot accuracy on DataComp medium, about 2% above prior best results and 5.5% above the best public-resource result.
- M-FLYT with SCS reaches 37.7% average accuracy over 38 DataComp tasks, outperforming the best prior public-resource approach by 0.4%.
- On DataComp small, M-FLYT + SCS achieves 8.0% ImageNet and 18.5% average accuracy, outperforming prior methods on both.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SCS (α = 0.15) instead of threshold top 20% sampling | M-FLYT · DataComp medium | ImageNet accuracy: 0.359 -> 0.401 | Helped |
| Apply SCS (α = 0.15) instead of threshold top 20% sampling | M-FLYT · DataComp medium | Average accuracy: 0.371 -> 0.377 | Helped |
| Apply SCS (α = 0.15) instead of threshold top 20% sampling | IN-weighted · DataComp medium | Average accuracy: 0.365 -> 0.362 | Hurt |
| Replace direct HYPE score summation with a learned M-FLYT mixture | HYPE component scores · DataComp medium | ImageNet accuracy: 0.309 -> 0.322 | Helped |
| Replace direct HYPE score summation with a learned M-FLYT mixture | HYPE component scores · DataComp medium | Average accuracy: 0.341 -> 0.347 | Helped |
| Use CE + temperature instead of CE downstream loss (ablation) | M-FLYT · DataComp medium | ImageNet accuracy: 0.359 -> 0.352 | Hurt |
| Use CE + temperature instead of CE downstream loss (ablation) | M-FLYT · DataComp medium | Average accuracy: 0.371 -> 0.366 | Hurt |
| Initialize the reference model with DataComp medium rather than random initialization (ablation) | M-FLYT · DataComp medium | ImageNet accuracy: 0.341 -> 0.359 | Helped |
| Initialize the reference model with DataComp medium rather than OpenAI (ablation) | M-FLYT · DataComp medium | ImageNet accuracy: 0.342 -> 0.359 | Helped |
| Remove input standardization (ablation) | M-FLYT · DataComp medium | ImageNet accuracy: 0.359 -> 0.323 | Hurt |
| Remove input standardization (ablation) | M-FLYT · DataComp medium | Average accuracy: 0.371 -> 0.337 | Hurt |
| Remove the DFN score group (ablation) | M-FLYT · DataComp medium | ImageNet accuracy: 0.359 -> 0.341 | Hurt |
Surprising findings
- Removing the linear binary classifiers, which are the weakest standalone filters, causes the largest average-accuracy drop (1.6%) in M-FLYT.
- E-FLYT, a learned scorer on CLIP embeddings, does not beat state-of-the-art filters on its own, though its scores improve M-FLYT when included.
Limitations
- Only tested at DataComp medium and small scales; larger scales untested.
- E-FLYT underperforms methods such as DFN, so M-FLYT relies on existing filters as inputs.
- Uses data weighting as a proxy for data selection and relies on ImageNet as downstream data.
Experimental setup
Datasets: DataComp medium ImageNet train set CC12M MS COCO Flickr30k CC2M
Benchmarks: DataComp medium DataComp small
Metrics: ImageNet zero-shot accuracy ImageNet distribution shifts accuracy VTAB accuracy retrieval accuracy average accuracy over 38 tasks
Baselines: DFN HYPE s-CLIPLoss HYPE+DFN+s-CLIPLoss EcoDatum WS CAM ∩ Flipped CLIP MGD
Builds on: DataComp CLIP FLYP HYPE s-CLIPLoss
Closest papers by meaning
- Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning (CVPR 2025 · similarity 0.87)
- Data Filtering Networks (ICLR 2024 · similarity 0.87)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.86)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.86)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.86)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.