Research map / Vision-language and multimodal / CLIP and zero-shot
Open-vocabulary alignment from VLMs
Refines vision-language alignment and similarity scoring to enable open-vocabulary recognition and segmentation.
130 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in CLIP and zero-shot, keyword cluster “zero shot · open · vocabulary”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 2 tracked benchmarks, but the best result has barely moved since 2024.
49 papers in 2023–24 and 66 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 13.2% to 13.2%.
What looks settled
- ADE20K (mIoU): best up to 2024 38.5, best in 2025–26 25.1 from 3 papers, at most 0.2% better, if at all.
- Cityscapes (mIoU): best up to 2024 61.25, best in 2025–26 41 from 3 papers, at most 0.2% better, if at all.
No longer reported by papers here: Food101 (accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Matching Networks for One Shot Learning (NeurIPS 2016 · 923 citations)
- 2017 · Link the Head to the "Beak": Zero Shot Learning From Noisy Text Description at Part Precision (CVPR 2017 · 129 citations)
- 2018 · Mix and Match Networks: Encoder-Decoder Alignment for Zero-Pair Image Translation (CVPR 2018 · 30 citations)
- 2019 · Few-Shot Image and Sentence Matching via Gated Visual-Semantic Embedding (AAAI 2019 · 30 citations)
- 2021 · Learning Transferable Visual Models From Natural Language Supervision (ICML 2021 · 5,285 citations)
- 2022 · GLIPv2: Unifying Localization and Vision-Language Understanding (NeurIPS 2022 · 126 citations)
- 2023 · ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding (CVPR 2023 · 241 citations)
- 2024 · Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (CVPR 2024 · 264 citations)
- 2025 · DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · 32 citations)
- 2026 · InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer (AAAI 2026 · 3 citations)
Earliest papers here
- Matching Networks for One Shot Learning (NeurIPS 2016 · 923 citations)
- Fast Zero-Shot Image Tagging (CVPR 2016 · 75 citations)
- Link the Head to the "Beak": Zero Shot Learning From Noisy Text Description at Part Precision (CVPR 2017 · 129 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ADE20K · mIoU (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 38.5 | Renovating Names in Open-Vocabulary Segmentation Benchmarks (NeurIPS 2024 · 1 citation) |
Cityscapes · mIoU (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 61.25 | Renovating Names in Open-Vocabulary Segmentation Benchmarks (NeurIPS 2024 · 1 citation) |
Food101 · accuracy (higher is better)
4 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 95.9 | Learning Transferable Visual Models From Natural Language Supervision (ICML 2021 · 5,285 citations) |
What has been tried
80 papers here report 945 measured changes (a component added, removed or swapped, compared with a baseline): helped 76%, no clear effect 5%, hurt 17%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace feature aggregation with cost aggregation and fine-tune CLIP | A-847 · mIoU | 5.6 -> 14.7 | Helped | CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · 135 citations) |
| Add 2 million SA1B images and 5 million GRIT examples to GLEE-Lite training | OmniLabel · AP | 20.3 -> 22.7 | Helped | General Object Foundation Model for Images and Videos at Scale (CVPR 2024 · 51 citations) |
| Add data augmentation | K=1, K=2, and K=4 · accuracy | 57.72(−3.5)↓; 61.94(−2.9)↓; 65.41(−2.5)↓ | Helped | A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · 45 citations) |
| add TED-Adapter | K400 · Top-1 accuracy | 56.5 -> 80.5 | Helped | A Multimodal, Multi-Task Adapting Framework for Video Action Recognition (AAAI 2024 · 38 citations) |
| Concatenate [CLS] with average-pooled patch features rather than using [CLS] alone | ImageNet-1K · accuracy | 78.8 -> 79.2 | Helped | DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · 32 citations) |
| Add fine-grained labels | TCGA Lung Cancer · ACC | 89.77 -> 91.21 | Helped | Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction (CVPR 2024 · 30 citations) |
| add YT-Storyboard-1B to pretraining | MS-COCO captioning · CIDEr | 110.8 -> 112.9 | Helped | Emu: Generative Pretraining in Multimodality (ICLR 2024 · 28 citations) |
| Add stable diffusion features F_sd to the baseline feature representation | HICO-DET · Full mAP | 31.99 -> 32.92 | Helped | Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · 25 citations) |
| Add orthogonal loss | Office-Home (average) · accuracy | 77.4 -> 77.9 | Helped | Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation (CVPR 2024 · 20 citations) |
| Use width 3968 and depth 32 (variant 1) instead of the selected variant 3 | zero-shot ImageNet variants · accuracy | 66.2 -> 65.8 | Hurt | InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (CVPR 2024 · 19 citations) |
| Add Document-former | CORD · eF1 | 38.5 -> 46.0 | Helped | InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · 18 citations) |
| add GradCAM sharpening | Pascal Context and COCO Stuff · mIoU | Helped | Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · 18 citations) |
What reviewers will expect
Datasets and baselines named in the 25 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO-Stuff · 6 of 25 papers
- ADE20K · 3 of 25 papers
- Cityscapes · 3 of 25 papers
- PASCAL VOC20 · 3 of 25 papers
- CC3M · 3 of 25 papers
- Food-101 · 3 of 25 papers
- EuroSAT · 2 of 25 papers
- COCO · 2 of 25 papers
Baselines
- CLIP · 4 of 25 papers
- SigLIP · 3 of 25 papers
- OpenCLIP · 2 of 25 papers
- TCL · 2 of 25 papers
- CoDe · 2 of 25 papers
- CLIP-DINOiser · 2 of 25 papers
- MaskCLIP · 2 of 25 papers
- CLIP ViT-L/14 · 2 of 25 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 115 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 5 papers
- Tencent · 4 papers
- Shanghai Jiao Tong University · 4 papers
- Huawei · 4 papers
- University of Tuebingen · 4 papers
- Microsoft · 3 papers
- The University of Hong Kong · 3 papers
- ByteDance · 3 papers
- Google · 3 papers
- Johns Hopkins University · 3 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Attribute-based zero-shot learning (115 papers)
- Adapting VLMs for zero-shot classification (108 papers)
- Language models for zero-shot reasoning (64 papers)
Elsewhere in the topic
- Prompt learning for CLIP (102 papers)
- Test-time adaptation of VLMs (93 papers)
- Recaptioning and text-supervised pretraining (73 papers)
- CLIP training recipes and data curation (72 papers)
- Fine-grained improvements to CLIP (63 papers)
- Feature calibration for unseen-class detection (50 papers)
- Self-distillation for vision-language pretraining (48 papers)
- Parameter-efficient visual tuning (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).