Research map / Vision-language and multimodal
CLIP and zero-shot: research map
1,294 papers on CLIP and zero-shot in Vision-language and multimodal, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 3 clusters and 12 approaches. The busiest year so far is 2026.
What it is
CLIP-style models learn from image-text pairs to put images and text in a shared space, so they can classify images into categories they were never trained on just from the category names. Research improves CLIP's training data and recipes and adapts it to new tasks with prompts or small adapters.
Terms you will see on this page:
- CLIP
- a model trained to match images with their captions by contrastive learning.
- Zero-shot classification
- classifying by comparing an image with text descriptions of the classes.
- Prompt learning
- learning the text prompt (or extra tokens) given to CLIP for a task.
Start reading with the most cited papers here:
- Learning Transferable Visual Models From Natural Language Supervision (ICML 2021 · 5,285 citations)
- Conditional Prompt Learning for Vision-Language Models (CVPR 2022 · 1,735 citations)
Within Vision-language and multimodal, its share shrank from 23.4% in 2023–24 to 10.5% in 2025–26 (371 → 500 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore CLIP and zero-shot in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Test-time adaptation of VLMs rising: 5% → 14% of the topic (20 → 71 papers)
- Prompt learning for CLIP falling: 13% → 10% of the topic (49 → 48 papers)
- Recaptioning and text-supervised pretraining falling: 9% → 6% of the topic (35 → 30 papers)
Papers from 2025–2026 that are already cited most:
- AA-CLIP: Enhancing Zero-Shot Anomaly Detection via Anomaly-Aware CLIP (CVPR 2025 · 69 citations)
- Enhance Vision-Language Alignment with Noise (AAAI 2025 · 42 citations)
- BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models (CVPR 2025 · 26 citations)
- CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · 19 citations)
Approaches and key papers
Zero-shot and open-vocabulary recognition · 567 papers
Keywords: zero shot, open, vocabulary
Approaches in this cluster:
- Open-vocabulary alignment from VLMs (165 papers)
Refines vision-language alignment and similarity scoring to enable open-vocabulary recognition and segmentation. - Attribute-based zero-shot learning (182 papers)
Uses attribute-guided transformers, prototypes and semantic grounding to classify unseen classes. - Adapting VLMs for zero-shot classification (146 papers)
Improves zero-shot classification by adding text descriptions, LLM prompts and long-tail handling to VLMs. - Language models for zero-shot reasoning (74 papers)
Leverages pretrained language models as priors or components for zero-shot multimodal and cross-lingual tasks.
Most cited and most cited since 2024:
- Learning Transferable Visual Models From Natural Language Supervision (ICML 2021 · 5,285 citations)
- Matching Networks for One Shot Learning (NeurIPS 2016 · 923 citations)
- AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models (AAAI 2024 · 279 citations)
- Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (CVPR 2024 · 264 citations)
Prompt and adapter tuning · 378 papers
Keywords: prompt, tuning, pre trained
Approaches in this cluster:
- Prompt learning for CLIP (144 papers)
Learns soft text prompts for vision-language models, with distillation, regularization and Bayesian modeling. - Test-time adaptation of VLMs (119 papers)
Adapts vision-language models on unlabeled test data via prompt tuning and lightweight adapters. - Feature calibration for unseen-class detection (57 papers)
Calibrates multimodal features to handle few-shot, unseen-class and out-of-distribution inputs. - Parameter-efficient visual tuning (58 papers)
Adapts pretrained vision and multimodal models through visual prompt tuning or efficient fine-tuning.
Most cited and most cited since 2024:
- Conditional Prompt Learning for Vision-Language Models (CVPR 2022 · 1,735 citations)
- MaPLe: Multi-Modal Prompt Learning (CVPR 2023 · 886 citations)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · 106 citations)
- MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (AAAI 2024 · 79 citations)
CLIP training and data · 349 papers
Keywords: contrastive, language image, image text
Approaches in this cluster:
- CLIP training recipes and data curation (86 papers)
Improves contrastive language-image pretraining through data scaling, curation and added supervision. - Recaptioning and text-supervised pretraining (109 papers)
Uses synthetic captions and interleaved image-text data to learn better visual representations. - Fine-grained improvements to CLIP (79 papers)
Strengthens CLIP's local detail understanding and studies its scaling behaviour. - Self-distillation for vision-language pretraining (75 papers)
Pretrains vision-language models with bootstrapping, masked and self-distillation objectives.
Most cited and most cited since 2024:
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (ICML 2021 · 1,195 citations)
- LAION-5B: An open large-scale dataset for training next generation image-text models (NeurIPS 2022 · 1,032 citations)
- VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection (AAAI 2024 · 206 citations)
- An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · 99 citations)
Related topics in Vision-language and multimodal
- Human and embodied modeling (2,095)
- Visual question answering (711)
- Multimodal LLMs (3,132)
- Robot action and navigation (841)
- Cross-modal learning (1,766)
