Research map / Vision-language and multimodal / CLIP and zero-shot
Adapting VLMs for zero-shot classification
Improves zero-shot classification by adding text descriptions, LLM prompts and long-tail handling to VLMs.
108 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in CLIP and zero-shot, keyword cluster “zero shot · open · vocabulary”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 3 tracked benchmarks.
47 papers in 2023–24 and 57 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 12.7% to 11.4%.
Open questions the data points to
- ImageNet (accuracy): best went from 77 (up to 2024) to 79.26 in 2025 — KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · 7 citations)
- Places365 (accuracy): best went from 44.09 (up to 2024) to 45.59 in 2025 — KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · 7 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · Multimodal Few-Shot Learning with Frozen Language Models (NeurIPS 2021 · 86 citations)
- 2022 · SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (ICLR 2022 · 338 citations)
- 2023 · Bidirectional Cross-Modal Knowledge Exploration for Video Recognition With Pre-Trained Vision-Language Models (CVPR 2023 · 119 citations)
- 2024 · AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models (AAAI 2024 · 279 citations)
- 2025 · SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · 26 citations)
- 2026 · RSVG-ZeroOV: Exploring a Training-Free Framework for Zero-Shot Open-Vocabulary Visual Grounding in Remote Sensing Images (AAAI 2026 · 5 citations)
Earliest papers here
- Multimodal Few-Shot Learning with Frozen Language Models (NeurIPS 2021 · 86 citations)
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (ICLR 2022 · 338 citations)
- Uni-Perceiver: Pre-Training Unified Architecture for Generic Perception for Zero-Shot and Few-Shot Tasks (CVPR 2022 · 74 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ImageNet · accuracy (higher is better)
7 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 75 | Visual Classification via Description from Large Language Models (ICLR 2023 · 56 citations) |
| 2024 | 77 | Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification (ICLR 2024 · 1 citation) |
| 2025 | 79.26 | KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · 7 citations) |
ImageNetV2 · accuracy (higher is better)
4 papers, 2023–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 69.3 | Visual Classification via Description from Large Language Models (ICLR 2023 · 56 citations) |
| 2024 | 71.05 | Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification (ICLR 2024 · 1 citation) |
Places365 · accuracy (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 40.55 | Visual Classification via Description from Large Language Models (ICLR 2023 · 56 citations) |
| 2024 | 44.09 | Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification (ICLR 2024 · 1 citation) |
| 2025 | 45.59 | KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · 7 citations) |
What has been tried
77 papers here report 961 measured changes (a component added, removed or swapped, compared with a baseline): helped 75%, no clear effect 5%, hurt 18%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add the lightweight feature-matching decoder | MVTec-AD (unsupervised) · Pixel-AUC | from ~72.2 accuracy-only baseline to 90.9 pixel-AUC with decoder | Helped | AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models (AAAI 2024 · 279 citations) |
| fine-tune the LLM during visual-language pre-training instead of freezing it | VLM ablation average (4 tasks), 4-shot · average accuracy/score | 44.5 -> 70.9 | Helped | VILA: On Pre-training for Visual Language Models (CVPR 2024 · 224 citations) |
| adapt CLIP with multi-level feature adapters rather than independently optimized multi-level projectors | Medical zero-shot benchmarks · AC AUC | +13.57% average improvement | Helped | Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images (CVPR 2024 · 113 citations) |
| Increase the retained SkyScript fraction from 20% to 50% | Seven scene-classification benchmarks · average top-1 accuracy (%) | 59.81 -> 59.93 | Helped | SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · 91 citations) |
| Add L2 normalization and adaptive temperature to the qkv self-self attention ensemble | PascalVOC · mIoU | 43.1 -> 45.1 | Helped | Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · 50 citations) |
| Use dynamic and static memory networks together | training-free few-shot classification · accuracy | Helped | Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models (CVPR 2024 · 34 citations) | |
| Add habitat descriptions to visual attributes | CUB, zero-shot test classes · accuracy | 53.34 -> 54.01 | Helped | Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · 33 citations) |
| Use visual programming instead of dialog with LLM | ScanRefer validation · Acc@0.5 | 25.4 -> 32.1 | Helped | Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding (CVPR 2024 · 32 citations) |
| use synonyms as well as original concept names for retrieval, rather than original names alone | standard zero-shot recognition datasets · mean per-class accuracy | +4% | Helped | The Neglected Tails in Vision-Language Models (CVPR 2024 · 27 citations) |
| add scene layout to 3D positions | Nr3D validation, Overall · accuracy | 37.7 -> 39.7 | Helped | SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · 26 citations) |
| Use a cleaned D200 dictionary instead of uncleaned D500 | CRC100K · weighted F1 | 0.804 -> 0.844 | Helped | CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment (CVPR 2024 · 26 citations) |
| Add Sky-SA dataset for fine-tuning | iSAID · mIoU | 18.40 -> 27.76 | Helped | SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling (CVPR 2025 · 25 citations) |
What reviewers will expect
Datasets and baselines named in the 30 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet · 6 of 30 papers
- DTD · 5 of 30 papers
- ImageNetV2 · 4 of 30 papers
- EuroSAT · 4 of 30 papers
- Oxford Pets · 4 of 30 papers
- Stanford Cars · 4 of 30 papers
- Food101 · 4 of 30 papers
- Flowers102 · 3 of 30 papers
Baselines
- CLIP · 5 of 30 papers
- CoOp · 3 of 30 papers
- VideoChatGPT · 3 of 30 papers
- TDA · 3 of 30 papers
- CLIP-Adapter · 2 of 30 papers
- YOLOv8 · 2 of 30 papers
- ZLaP · 2 of 30 papers
- DMN · 2 of 30 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 104 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 8 papers
- National University of Singapore · 6 papers
- Stanford University · 5 papers
- Nanjing University · 4 papers
- Meta · 4 papers
- A*STAR · 4 papers
- Nanyang Technological University · 4 papers
- Carnegie Mellon University · 3 papers
- Hong Kong University of Science and Technology · 3 papers
- MIT-IBM Watson AI Lab · 3 papers
Use it
Download BibTeX of 31 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Open-vocabulary alignment from VLMs (130 papers)
- Attribute-based zero-shot learning (115 papers)
- Language models for zero-shot reasoning (64 papers)
Elsewhere in the topic
- Prompt learning for CLIP (102 papers)
- Test-time adaptation of VLMs (93 papers)
- Recaptioning and text-supervised pretraining (73 papers)
- CLIP training recipes and data curation (72 papers)
- Fine-grained improvements to CLIP (63 papers)
- Feature calibration for unseen-class detection (50 papers)
- Self-distillation for vision-language pretraining (48 papers)
- Parameter-efficient visual tuning (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).