Research map / Vision-language and multimodal / CLIP and zero-shot
CLIP training recipes and data curation
Improves contrastive language-image pretraining through data scaling, curation and added supervision.
72 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2022–2026) · an approach in CLIP and zero-shot, keyword cluster “contrastive · language image · image text”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
27 papers in 2023–24 and 40 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 7.3% to 8.0%.
Open questions the data points to
- Papers report CLIP on CIFAR100 zero-shot (top-1 accuracy) anywhere from 18.69 to 48.6: comparisons against it depend on whose number you take.
- Papers report CLIP on ImageNetV2 zero-shot (top-1 accuracy) anywhere from 16.91 to 32.8: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2022 · How Much Can CLIP Benefit Vision-and-Language Tasks? (ICLR 2022 · 150 citations)
- 2023 · CALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention (AAAI 2023 · 117 citations)
- 2024 · An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · 99 citations)
- 2025 · CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination (AAAI 2025 · 16 citations)
- 2026 · HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment (AAAI 2026 · 3 citations)
Earliest papers here
- How Much Can CLIP Benefit Vision-and-Language Tasks? (ICLR 2022 · 150 citations)
- Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (ICLR 2022 · 128 citations)
- CyCLIP: Cyclic Contrastive Language-Image Pretraining (NeurIPS 2022 · 46 citations)
What has been tried
54 papers here report 678 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 8%, hurt 21%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add the four training tricks: global gradients, dropout, locked bottom layer, and soft label | CUHK-PEDES · Rank-1 | 60.67 -> 64.34 | Helped | An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · 99 citations) |
| Change balancing threshold from t=20k to t=15k | ImageNet zero-shot · accuracy | 65.5 -> 65.5 | No clear effect | Demystifying CLIP Data (ICLR 2024 · 22 citations) |
| add text-guided image colorization (TIC) | CUHK-PEDES (zero-shot; ablation subset) · R@1 | 30.2 -> 32.5 | Helped | PLIP: Language-Image Pre-training for Person Representation Learning (NeurIPS 2024 · 20 citations) |
| Add regional contrastive loss to global contrastive learning | COCO val2017 box classification · Top1 mean accuracy | 42.3 -> 47.8 | Helped | FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · 15 citations) |
| Add intra-modal contrast to hard-negative image-text contrastive training | ARO Relation · accuracy | 81.6 -> 82.6 | Helped | Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · 14 citations) |
| Use a Polynomial kernel instead of an RBF kernel | 11 classification datasets, CoOp benchmark, 16 shots · accuracy | 76.75 -> 76.61 | Hurt | ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models (CVPR 2025 · 12 citations) |
| add noise injection | MSCOCO · CIDEr | 0.221 -> 0.286 | Helped | Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · 11 citations) |
| Add ATRM to CLIM, refining image tokens | Urban1k · T2I Recall@1 | 0.782 -> 0.890 | Helped | FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · 10 citations) |
| Add TGI and IGT to GDA-CLIP | 11-dataset average · top-1 accuracy | 69.87 -> 71.19 | Helped | Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP (AAAI 2025 · 7 citations) |
| Use both negative captions and negative images rather than negative captions alone | SugarCrepe (CC3M) · accuracy | 61.69 -> 63.49 | Helped | TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · 7 citations) |
| add RaFA to HyCD | Zero-shot classification (12 datasets average) · top-1 accuracy | 53.13 -> 54.69 | Helped | Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · 6 citations) |
| scale seen training pairs from 1.0× to 2.3× for worldwide data | ImageNet val · zero-shot accuracy | 79.5 -> 81.3 | Helped | Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · 5 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| CLIP | CIFAR100 zero-shot | top-1 accuracy | 18.69 – 48.6 | 2 papers, e.g. NeurIPS 2022 and CVPR 2023 |
| CLIP | ImageNetV2 zero-shot | top-1 accuracy | 16.91 – 32.8 | 2 papers, e.g. NeurIPS 2022 and CVPR 2023 |
| CLIP | CIFAR10 zero-shot | top-1 accuracy | 46.54 – 76 | 2 papers, e.g. NeurIPS 2022 and CVPR 2023 |
What reviewers will expect
Datasets and baselines named in the 28 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet · 7 of 28 papers
- CC3M · 7 of 28 papers
- CC12M · 5 of 28 papers
- ShareGPT4V · 5 of 28 papers
- COCO · 5 of 28 papers
- Flickr30K · 4 of 28 papers
- MSCOCO · 3 of 28 papers
- ImageNet-1K · 3 of 28 papers
Baselines
- CLIP · 8 of 28 papers
- Long-CLIP · 6 of 28 papers
- CoOp · 4 of 28 papers
- SigLIP · 3 of 28 papers
- CLIP-Adapter · 3 of 28 papers
- Tip-Adapter · 2 of 28 papers
- APE · 2 of 28 papers
- Tip-Adapter-F · 2 of 28 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 67 papers; from OpenAlex and the papers' affiliations).
- Huazhong University of Science and Technology · 4 papers
- Meta · 4 papers
- Carnegie Mellon University · 3 papers
- ShanghaiTech University · 3 papers
- ByteDance · 3 papers
- Microsoft · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Google · 3 papers
- Shanghai AI Laboratory · 2 papers
- Peking University · 2 papers
Use it
Download BibTeX of 33 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Recaptioning and text-supervised pretraining (73 papers)
- Fine-grained improvements to CLIP (63 papers)
- Self-distillation for vision-language pretraining (48 papers)
Elsewhere in the topic
- Open-vocabulary alignment from VLMs (130 papers)
- Attribute-based zero-shot learning (115 papers)
- Adapting VLMs for zero-shot classification (108 papers)
- Prompt learning for CLIP (102 papers)
- Test-time adaptation of VLMs (93 papers)
- Language models for zero-shot reasoning (64 papers)
- Feature calibration for unseen-class detection (50 papers)
- Parameter-efficient visual tuning (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).