Research map / Vision-language and multimodal / CLIP and zero-shot
Language models for zero-shot reasoning
Leverages pretrained language models as priors or components for zero-shot multimodal and cross-lingual tasks.
64 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in CLIP and zero-shot, keyword cluster “zero shot · open · vocabulary”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 28 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 5.7% to 5.6%.
Open questions the data points to
- Papers report CLAP on FSD50K zero-shot (MAP) anywhere from 0.14 to 0.302: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Neural Cross-Lingual Entity Linking (AAAI 2018 · 67 citations)
- 2020 · Cross-Lingual Natural Language Generation via Pre-Training (AAAI 2020 · 139 citations)
- 2021 · Multilingual Transfer Learning for QA using Translation as Data Augmentation (AAAI 2021 · 31 citations)
- 2022 · Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification (AAAI 2022 · 66 citations)
- 2023 · InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (NeurIPS 2023 · 409 citations)
- 2024 · ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding (CVPR 2024 · 122 citations)
- 2025 · CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · 19 citations)
- 2026 · TIM++: Transductive Information Maximization for Few-Shot CLIP (AAAI 2026 · 2 citations)
Earliest papers here
- Neural Cross-Lingual Entity Linking (AAAI 2018 · 67 citations)
- Cross-Lingual Natural Language Generation via Pre-Training (AAAI 2020 · 139 citations)
- Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems (AAAI 2020 · 89 citations)
What has been tried
51 papers here report 610 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 6%, hurt 17%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace manual captions with top-1 holistic BLIP-2 captions | ModelNet40, zero-shot · top-1 accuracy | 60.4 -> 69.7 | Helped | ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding (CVPR 2024 · 122 citations) |
| Add language-only pre-training | M-BEIR test set · average recall performance | 53.6 -> 56.6 | Helped | LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · 16 citations) |
| Add QPMIL to FineTune | TCGA WSI, forward-order class-incremental · ACC | 0.308 ± 0.045 -> 0.720 ± 0.029 | Helped | Queryable Prototype Multiple Instance Learning with Vision-Language Models for Incremental Whole Slide Image Classification (AAAI 2025 · 13 citations) |
| add Chinese image-text pairs during pretraining | LLaVA Test Set, Chinese · GPT-4 AVG score | 90.9 -> 92.5 | Helped | Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages (ICLR 2024 · 13 citations) |
| use lower top-p sampling thresholds | VALL-E zero-shot TTS decoding · WER (%) | 5.47 -> 49.26 | Hurt | ELLA-V: Stable Neural Codec Language Modeling with Alignment-Guided Sequence Reordering (AAAI 2025 · 10 citations) |
| Increase timbre prompt length from 10s to 60s or 300s | LibriSpeech test-clean · SIM | 0.905 -> 0.922; 0.905 -> 0.930 | Helped | Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · 8 citations) |
| Combine PKI and LoRA in PLoRA | IMDB-A full-shot · F1 | 48.9 -> 54.9 | Helped | Personalized LoRA for Human-Centered Text Understanding (AAAI 2024 · 7 citations) |
| pre-train with Wikiformer's four Wikipedia-structure tasks instead of BERT-base alone | MS MARCO zero-shot · MRR@10 | 0.1684 -> 0.2844 | Helped | Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-Hoc Retrieval (AAAI 2024 · 5 citations) |
| pre-train with cell-text alignment and matching | PBMC10K zero-shot cell type annotation · accuracy | 85.8 -> 86.5 | Helped | LangCell: Language-Cell Pre-training for Cell Identity Understanding (ICML 2024 · 5 citations) |
| Use bidirectional alignment with MIL-InfoNCE instead of the tested alternative positive-sample aggregations | EEG retrieval, Top-10 · accuracy | 3.9, 8.3, 24.7 -> 27.1 | Helped | EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping (ICML 2025 · 2 citations) |
| Use structural and semantic captions instead of semantic captions alone | Activity zero-shot classification · AUROC | 0.71 -> 0.84 | Helped | SensorLM: Learning the Language of Wearable Sensors (NeurIPS 2025 · 2 citations) |
| Use compute-matched rather than full-compute echo embeddings | MTEB (56 datasets), zero-shot · average | 48.64 -> 49.02 | No clear effect | Repetition Improves Language Model Embeddings (ICLR 2025 · 2 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| CLAP | FSD50K zero-shot | MAP | 0.14 – 0.302 | 2 papers, e.g. NeurIPS 2023 and NeurIPS 2023 |
| CLAP | ESC-50 zero-shot | accuracy | 0.77 – 0.826 | 2 papers, e.g. NeurIPS 2023 and NeurIPS 2023 |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- SimCLR · 2 of 19 papers
- BYOL · 2 of 19 papers
- CLIP · 2 of 19 papers
- GritLM-7B · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 49 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 4 papers
- Chinese Academy of Sciences · 4 papers
- ByteDance · 3 papers
- Google · 3 papers
- Tsinghua University · 3 papers
- University of Chinese Academy of Sciences · 2 papers
- Shanghai Jiao Tong University · 2 papers
- Stanford University · 2 papers
- University of Texas at Austin · 2 papers
- Zhejiang University · 2 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Open-vocabulary alignment from VLMs (130 papers)
- Attribute-based zero-shot learning (115 papers)
- Adapting VLMs for zero-shot classification (108 papers)
Elsewhere in the topic
- Prompt learning for CLIP (102 papers)
- Test-time adaptation of VLMs (93 papers)
- Recaptioning and text-supervised pretraining (73 papers)
- CLIP training recipes and data curation (72 papers)
- Fine-grained improvements to CLIP (63 papers)
- Feature calibration for unseen-class detection (50 papers)
- Self-distillation for vision-language pretraining (48 papers)
- Parameter-efficient visual tuning (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).