Research map / Vision-language and multimodal / CLIP and zero-shot
Data Filtering Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that data filtering networks (DFNs), CLIP-style models trained on small high-quality data, can filter large web image-text pools into state-of-the-art training sets (DFN-2B, DFN-5B) for CLIP models.
Problem
Filtering a large uncurated web image-text pool into a high-quality pre-training set; it is unclear which properties of a model make it a good filter, since ImageNet accuracy does not predict filtering quality.
Approach
Treat a filtering model as a distinct object: a CLIP model trained on high-quality data (fine-tuned with weight ensembling and ImageNet/COCO/Flickr data) applied pointwise to a raw pool; its filtering quality is measured by the performance of models trained on the induced dataset rather than its own downstream accuracy.
Key results
- ViT-H/14 trained on DFN-5B reaches 84.4% zero-shot ImageNet accuracy and 0.710 DataComp average.
- ViT-L/14 trained on DFN-2B for 12.8B samples seen reaches 81.4% ImageNet zero-shot, a 2.2-point gain over DataComp-1B's ViT-L.
- A ViT-L/14 on DFN-2B surpasses a ViT-G/14 trained on LAION-2B by 1.5% ImageNet zero-shot accuracy with about 16x less compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the filtering network on high-quality CC12M rather than ImageNet-labeled data | CLIP ViT-B/32 data filtering network · DataComp medium induced dataset | ImageNet top-1: 0.266 -> 0.289 | Helped |
| Use CLIP filtering rather than an ImageNet binary filter | Data filtering network · DataComp medium induced dataset | Average: 0.295 -> 0.335 | Helped |
| Add augmentation (ablation) | DFN training · DataComp large | Average: 0.536 -> 0.542 | Helped |
| Increase samples seen and batch size from 2.56B / 4096 to 5.12B / 8192 (ablation) | DFN training · DataComp large | Average: 0.545 -> 0.550 | Helped |
| Add fine-tuning (ablation) | DFN training · DataComp large | Average: 0.550 -> 0.560 | Helped |
| Use OpenAI initialization (ablation) | DFN training · DataComp large | Average: 0.548 -> 0.560 | Helped |
| Train the filtering network on HQITP-350M and use its induced data rather than adding HQITP-350M to the OpenAI-filtered dataset | ViT-L/14 CLIP training · DataComp evaluation | Average: 0.654 -> 0.669 | Helped |
| Add HQITP-350M directly to DFN-2B training data (ablation) | ViT-L/14 CLIP training · DataComp evaluation | Average: 0.669 -> 0.670 | No clear effect |
| Fine-tune the DFN on ImageNet instead of directly adding ImageNet to training data | Baseline DFN · ImageNet distribution shifts | ObjectNet: 0.509 -> 0.536 | Helped |
| Fine-tune the DFN on ImageNet rather than directly training on ImageNet | Baseline DFN · ImageNet distribution shifts | IN-Sketch: 0.512 -> 0.536 | Helped |
| Fine-tune the DFN on ImageNet rather than directly training on ImageNet | Baseline DFN · ImageNet distribution shifts | IN-R: 0.703 -> 0.743 | Helped |
| Train the CLIP filtering network on publicly available CC12M + CC3M + SS15M instead of OpenAI WIT-400M | ViT-B/32 DFN · DataComp medium | Average: 0.338 -> 0.349 | Helped |
Surprising findings
- Filtering strength is uncorrelated with ImageNet performance: a CLIP model with about 30% lower ImageNet accuracy than OpenAI CLIP can be a better filter.
- Mixing even a small fraction of unfiltered Common Crawl data into the filter training pool sharply degrades filtering performance, while ImageNet accuracy of the filter declines gradually.
Limitations
- Filtering relies on CLIP image-text alignment as a weak proxy for dataset quality; the paper does not know how to directly optimize dataset quality.
- Applicability to other domains such as speech, text, or video is unclear.
- Gains from DFN fine-tuning on ImageNet and the training of the DFN depend on a high-quality human-verified dataset (HQITP-350M) that is not public.
Experimental setup
Datasets: HQITP-350M MS COCO Flickr30k ImageNet-1k DataComp CommonPool Conceptual 12M Conceptual Captions 3M Shutterstock 15M LAION-2B
Benchmarks: DataComp 38-task suite ImageNet ImageNet-V2 ObjectNet ImageNet-Sketch ImageNet-R ImageNet-A VQAv2 GQA OKVQA
Metrics: zero-shot top-1 accuracy average accuracy over 38 tasks retrieval recall
Baselines: LAION-2B DataComp-1B OpenAI WIT-400M MetaCLIP WebLI CLIPA-v2
Builds on: CLIP DataComp OpenCLIP BLIP-2 MetaCLIP
Closest papers by meaning
- A Bitter Lesson for Data Filtering (NeurIPS 2026 · similarity 0.88)
- Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · similarity 0.88)
- SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.87)
- Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality (NeurIPS 2025 · similarity 0.87)
- Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining (NeurIPS 2025 · similarity 0.87)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.