Research map / Vision-language and multimodal / CLIP and zero-shot
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that zero-shot performance of CLIP and text-to-image models tracks log-linearly with the pretraining frequency of downstream concepts, and introduces the long-tailed Let It Wag! benchmark on which models underperform.
Problem
Whether "zero-shot" generalization in multimodal models (CLIP classification/retrieval and Stable Diffusion T2I) is genuine, given that downstream concepts may be frequent in web-crawled pretraining data.
Approach
Estimate concept frequencies in pretraining image-text datasets by intersecting text unigram search with RAM++ image tagging, then show zero-shot performance scales log-linearly with concept frequency, implying exponential data needs for linear gains; further verified with similarity-pruned and synthetic pretraining data and a long-tailed test set.
Key results
- Across CLIP models and four pretraining datasets, zero-shot performance shows a consistent log-linear relationship with log-scaled concept frequency, with the trend persisting after pruning 50M similar samples from LAION-200M and on SynthCI-30M synthetic data.
- Up to 20% lower absolute accuracy on the Let It Wag! long-tailed classification set compared to ImageNet across 40 CLIP models.
- Pretraining datasets show strongly correlated concept distributions (e.g., CC-12M vs YFCC-15M r=0.97) and image-text misalignment of 5%-36% of pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase a concept's pretraining frequency | CLIP models · 17 classification datasets | zero-shot accuracy | Helped |
| Increase a concept's pretraining frequency | CLIP models · 2 retrieval datasets | Recall@10 | Helped |
| Increase a concept's pretraining frequency | Text-to-image models pretrained on LAION-Aesthetics · 8 text-to-image datasets | expected aesthetics score | Helped |
| Remove the 50 million most test-similar samples from pretraining data | CLIP ViT-B-32 pretrained on LAION-200M · Classification and retrieval tasks | frequency-performance relationship | No clear effect |
| Use a controlled synthetic pretraining distribution | Models pretrained on SynthCI-30M · Zero-shot performance across concepts | frequency-performance relationship | No clear effect |
| Evaluate on long-tail concepts rather than ImageNet concepts | CLIP models · Let It Wag! versus ImageNet | accuracy: upto 20% lower absolute accuracies | Hurt |
| Increase model capacity | CLIP models · Let It Wag! classification | performance gap to ImageNet: 1.58 | Helped |
| Use RAM++ threshold 0.7 instead of 0.5 or 0.6 (ablation) | RAM++ image concept tagging · Pretraining concept frequency estimation | frequency estimation precision | Helped |
| Lower the RAM++ threshold to 0.5 or 0.6 (ablation) | RAM++ image concept tagging · Pretraining concept frequency estimation | frequency estimation precision | Hurt |
| Use the prompt "A photo of a {classname}" or an 80-prompt ensemble instead of the classname alone (ablation) | CLIP models · Zero-shot classification | frequency-performance relationship | No clear effect |
| Use SLIP or CyCLIP training objectives (ablation) | CLIP models · Zero-shot classification and retrieval | frequency-performance relationship | No clear effect |
Surprising findings
- Pretraining concept distributions are highly correlated across very different datasets (3M to 400M samples), suggesting web crawls share the same long tail.
- Higher-capacity CLIP models show a steeper slope (1.58) on Let It Wag!, indicating they are closing the gap to ImageNet on long-tailed concepts.
Limitations
- Frequency is estimated via text unigram search and RAM++ tagging, which may be noisy and depends on a threshold of 0.7 chosen for precision.
- Let It Wag! images are sourced from the web and may overlap with pretraining data despite temporal filtering.
- Human study for T2I uses only 6 participants and a small sample of 360 people.
Experimental setup
Datasets: CC-3M CC-12M YFCC-15M LAION-Aesthetics LAION-400M LAION-200M ImageNet SUN397 UCF101 Caltech101 EuroSAT CUB
Benchmarks: ImageNet SUN397 UCF101 Caltech101 EuroSAT CUB Caltech256 Flowers102 DTD Birdsnap Food101 Stanford-Cars
Metrics: accuracy Recall@1 Recall@5 Recall@10 expected CLIP-score max CLIP-score expected aesthetics score max aesthetics score human alignment score Pearson correlation
Baselines: CLIP Stable Diffusion DeepFloyd-IF DALL-E Mini SD-XL
Builds on: CLIP RAM++ SynthCI-30M HEIM
Closest papers by meaning
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.89)
- On the Out-Of-Distribution Generalization of Large Multimodal Models (CVPR 2025 · similarity 0.88)
- ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (ICML 2026 · similarity 0.88)
- OTTER: Effortless Label Distribution Adaptation of Zero-shot Models (NeurIPS 2024 · similarity 0.87)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.87)
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.