Research map / Vision-language and multimodal / CLIP and zero-shot
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic
Yoad Tewel, Yoav Shalev, Idan Schwartz, Lior Wolf
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Guide GPT-2 by optimizing its context cache with CLIP image alignment and an LM-distribution regularizer | GPT-2 medium + CLIP ViT-B/32 · COCO test set | CLIP-S: 0.77 -> 0.87 | Improved |
| Guide GPT-2 by optimizing its context cache with CLIP image alignment and an LM-distribution regularizer | GPT-2 medium + CLIP ViT-B/32 · COCO test set | BLEU-4: 41.0 -> 2.6 | Hurt |
| Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioning | ZeroCap versus ClipCap · Visual Relations: Building → Country | Recall@5: 0.035 -> 0.32 | Improved |
| Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioning | ZeroCap versus ClipCap · Visual Relations: Country → Capital | Recall@5: 0.0 -> 0.32 | Improved |
| Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioning | ZeroCap versus ClipCap · Visual Relations: CEO → Company | Recall@5: 0.05 -> 0.3 | Improved |
| Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioning | ZeroCap versus ClipCap · Visual Relations: Food → Country | Recall@5: 0.0 -> 0.33 | Improved |
| Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioning | ZeroCap versus ClipCap · Visual Relations: Leader → Country | Recall@5: 0.24 -> 0.28 | Improved |
| Replace context-cache optimization with greedy token selection maximizing CLIP score (ablation) | ZeroCap · COCO validation set | Perplexity: 5.50 -> 8.61 | Hurt |
| Replace context-cache optimization with greedy token selection maximizing CLIP score (ablation) | ZeroCap · COCO validation set | CLIP-S: 0.87 -> 0.98 | Improved |
| Replace context-cache optimization with greedy selection using combined CLIP and LM scores (ablation) | ZeroCap · COCO validation set | Perplexity: 5.50 -> 6.04 | Hurt |
| Replace context-cache optimization with greedy selection using combined CLIP and LM scores (ablation) | ZeroCap · COCO validation set | CLIP-S: 0.87 -> 0.91 | Improved |
| Add the negative prior against capitalized subwords to suppress source-related or irrelevant named entities (ablation) | ZeroCap · Image captioning | Entity relevance | Improved |
Later papers that compared with this method
5 later accepted papers report ZeroCap as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.
- ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation (TPAMI 2024 · better on 1 of 1 shared result)
- ConZIC: Controllable Zero-Shot Image Captioning by Sampling-Based Polishing (CVPR 2023 · better on 6 of 7 shared results)
- DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training (ICLR 2023 · better on 2 of 2 shared results)
- Fusing Pre-Trained Language Models With Multimodal Prompts Through Reinforcement Learning (CVPR 2023 · better on 1 of 1 shared result)
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (ICLR 2023 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- MeaCap: Memory-Augmented Zero-shot Image Captioning (CVPR 2024 · similarity 0.89)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.89)
- Zero-Shot Image Captioning with Multi-type Entity Representations (AAAI 2025 · similarity 0.88)
- Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions (CVPR 2024 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
