Research map / Vision-language and multimodal / CLIP and zero-shot
Self-distillation for vision-language pretraining
Pretrains vision-language models with bootstrapping, masked and self-distillation objectives.
48 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2022–2026) · an approach in CLIP and zero-shot, keyword cluster “contrastive · language image · image text”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 20 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 5.7% to 4.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2022 · BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022 · 863 citations)
- 2023 · BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (ICML 2023 · 917 citations)
- 2024 · Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · 29 citations)
- 2025 · Enhance Vision-Language Alignment with Noise (AAAI 2025 · 42 citations)
- 2026 · Q-CLIP: Unleashing the Power of Vision-Language Models for Video Quality Assessment through Unified Cross-Modal Adaptation (ICML 2026 · 0 citations)
Earliest papers here
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022 · 863 citations)
- FILIP: Fine-grained Interactive Language-Image Pre-Training (ICLR 2022 · 205 citations)
- PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (NeurIPS 2022 · 42 citations)
What has been tried
43 papers here report 495 measured changes (a component added, removed or swapped, compared with a baseline): helped 73%, no clear effect 9%, hurt 15%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Inject visual noise into visual features using Cross Attention rather than image-level MLP | ImageNet, 16-shot · accuracy | 67.43 -> 70.93 | Helped | Enhance Vision-Language Alignment with Noise (AAAI 2025 · 42 citations) |
| Combine Align and Distill stages instead of aligning first | OH, TI, VLCS, and PACS average · OOD accuracy | 79.50 -> 73.24 | Hurt | Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · 29 citations) |
| Add augmentation | DataComp large · Average | 0.536 -> 0.542 | Helped | Data Filtering Networks (ICLR 2024 · 15 citations) |
| Use concept filtering rather than full-dataset continual pretraining | General bioimaging classification (35 tasks) · average accuracy | 41.35 -> 47.72 | Helped | BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature (CVPR 2025 · 14 citations) |
| Use curated 14M BLIP pretraining data instead of BLIP-129M | Average over 38 datasets · average score | 33.05 -> 34.03 | Helped | Sieve: Multimodal Dataset Pruning using Image Captioning Models (CVPR 2024 · 14 citations) |
| Add ECM to identify and use false negatives as positives | COCO image-text retrieval, TR R@1 · R@1 | 76.6 -> 77.4 | Helped | MAFA: Managing False Negatives for Vision-Language Pre-training (CVPR 2024 · 11 citations) |
| Add image augmentation | ImageNet · top-1 accuracy | 17.5 -> 19.1 | Helped | COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training (CVPR 2025 · 10 citations) |
| Add LCVTA without LMRTVA | CAM16 · weighted F1 | 0.567 -> 0.614 | Helped | Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation (CVPR 2025 · 7 citations) |
| Fix incorrect negatives using similarity-based positive-pair mining | ImageNet · Top-1 accuracy | 18.6 -> 21.3; 23.3 -> 26.8; 31.1 -> 32.9 | Helped | FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models (CVPR 2024 · 7 citations) |
| add masked distillation and teacher word-embedding projection | ImageNet-ZeroShot · accuracy | 31.5 -> 33.9 | Helped | Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · 6 citations) |
| Use identical-distribution initialization for new token embeddings | MSCOCO₃₆, image-to-text · average Recall@1 | 29.6 -> 32.4 | Helped | Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · 5 citations) |
| Add the counterfactual debiasing loss | FairFace gender debiasing · ABLE | 75.58 -> 78.35 | Helped | Joint Vision-Language Social Bias Removal for CLIP (CVPR 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 15 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet · 4 of 15 papers
- CC3M · 4 of 15 papers
- CC12M · 4 of 15 papers
- YFCC15M · 3 of 15 papers
- ImageNet-1k · 3 of 15 papers
- Flickr30k · 3 of 15 papers
- MSCOCO · 3 of 15 papers
- Caltech101 · 2 of 15 papers
Baselines
- CLIP · 5 of 15 papers
- DreamLIP · 3 of 15 papers
- OpenCLIP · 3 of 15 papers
- SigLIP · 3 of 15 papers
- MaskCLIP · 2 of 15 papers
- SCLIP · 2 of 15 papers
- CoCa · 2 of 15 papers
- PMC-CLIP · 2 of 15 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 41 papers; from OpenAlex and the papers' affiliations).
- University of Washington · 3 papers
- Adobe · 3 papers
- Stanford University · 3 papers
- Google · 3 papers
- Peking University · 2 papers
- LAION · 2 papers
- Juelich Supercomputing Center · 2 papers
- Meta · 2 papers
- Chinese Academy of Sciences · 2 papers
- Google DeepMind · 2 papers
Use it
Download BibTeX of 28 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- CLIP training recipes and data curation (72 papers)
- Recaptioning and text-supervised pretraining (73 papers)
- Fine-grained improvements to CLIP (63 papers)
Elsewhere in the topic
- Open-vocabulary alignment from VLMs (130 papers)
- Attribute-based zero-shot learning (115 papers)
- Adapting VLMs for zero-shot classification (108 papers)
- Prompt learning for CLIP (102 papers)
- Test-time adaptation of VLMs (93 papers)
- Language models for zero-shot reasoning (64 papers)
- Feature calibration for unseen-class detection (50 papers)
- Parameter-efficient visual tuning (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).