AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

DataComp: In search of the next generation of multimodal datasets

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari and 22 more

NeurIPS 2023 · Vision-language and multimodal › CLIP and zero-shot · approach: Recaptioning and text-supervised pretraining · 73 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
filter the pool with image-based intersect CLIP score instead of no filteringCLIP ViT-L/14 on CommonPool · ImageNet, xlargeaccuracy: 0.723 -> 0.792Improved
use CLIP score (L/14 30%) filtering instead of no filteringCLIP on CommonPool · ImageNet, xlargeaccuracy: 0.723 -> 0.764Improved
add image-based filtering on top of CLIP score filteringCLIP on CommonPool · ImageNet, largeaccuracy: 0.578 -> 0.631Improved
add image-based filtering on top of CLIP score filteringCLIP on CommonPool · Retrieval, smallaccuracy: 0.119 -> 0.094Hurt
train on DataComp-1B instead of OpenAI WIT, same computeCLIP ViT-L/14 · ImageNet zero-shotaccuracy (%): 75.5 -> 79.2Improved
train on DataComp-1B instead of LAION-2BCLIP ViT-L/14 · ImageNet zero-shotaccuracy (%): 73.1 -> 79.2Improved
add external BYOD data (CC12M, YFCC15M, RedCaps, Shutterstock) to CLIP-filtered poolCLIP at large scale · ImageNetaccuracy: +4.3 percentage pointsImproved
use random subsets of pool instead of full poolCLIP on CommonPool · ImageNetaccuracyHurt
keep top ~30% by CLIP score rather than other fractionsCLIP score filtering · ImageNetaccuracyImproved
apply LAION-style filtering on CommonPool versus LAION-2B itselfCLIP large scale · ImageNetaccuracy (%): 55.3 vs 55.7No clear effect
add face blurring to training images (ablation)CLIP medium scale, CLIP score (B/32, 30%) · ImageNetaccuracy: 0.287 -> 0.282No clear effect
add face blurring to training images (ablation)CLIP medium scale, CLIP score (B/32, thresh 0.3)+English · ImageNetaccuracy: 0.209 -> 0.196Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy