AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Yoad Tewel, Yoav Shalev, Idan Schwartz, Lior Wolf

CVPR 2022 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 142 citations (OpenAlex)

Check your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Guide GPT-2 by optimizing its context cache with CLIP image alignment and an LM-distribution regularizerGPT-2 medium + CLIP ViT-B/32 · COCO test setCLIP-S: 0.77 -> 0.87Improved
Guide GPT-2 by optimizing its context cache with CLIP image alignment and an LM-distribution regularizerGPT-2 medium + CLIP ViT-B/32 · COCO test setBLEU-4: 41.0 -> 2.6Hurt
Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioningZeroCap versus ClipCap · Visual Relations: Building → CountryRecall@5: 0.035 -> 0.32Improved
Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioningZeroCap versus ClipCap · Visual Relations: Country → CapitalRecall@5: 0.0 -> 0.32Improved
Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioningZeroCap versus ClipCap · Visual Relations: CEO → CompanyRecall@5: 0.05 -> 0.3Improved
Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioningZeroCap versus ClipCap · Visual Relations: Food → CountryRecall@5: 0.0 -> 0.33Improved
Use visual-semantic arithmetic and cache-guided generation instead of COCO-supervised captioningZeroCap versus ClipCap · Visual Relations: Leader → CountryRecall@5: 0.24 -> 0.28Improved
Replace context-cache optimization with greedy token selection maximizing CLIP score (ablation)ZeroCap · COCO validation setPerplexity: 5.50 -> 8.61Hurt
Replace context-cache optimization with greedy token selection maximizing CLIP score (ablation)ZeroCap · COCO validation setCLIP-S: 0.87 -> 0.98Improved
Replace context-cache optimization with greedy selection using combined CLIP and LM scores (ablation)ZeroCap · COCO validation setPerplexity: 5.50 -> 6.04Hurt
Replace context-cache optimization with greedy selection using combined CLIP and LM scores (ablation)ZeroCap · COCO validation setCLIP-S: 0.87 -> 0.91Improved
Add the negative prior against capitalized subwords to suppress source-related or irrelevant named entities (ablation)ZeroCap · Image captioningEntity relevanceImproved

Later papers that compared with this method

5 later accepted papers report ZeroCap as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy