Research map / Vision-language and multimodal / CLIP and zero-shot
Zero-Shot Image Captioning with Multi-type Entity Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MERCap, a text-only zero-shot image captioner that retrieves multi-representation entities trained by contrastive learning and feeds them as hard prompts to a GPT-2 decoder alongside CLIP soft prompts.
Problem
Zero-shot image captioning without paired image-text data, where entity retrieval recall limits caption quality and entities are typically given only a single representation.
Approach
Approximates image embeddings with CLIP text embeddings plus Gaussian noise, learns K representation vectors per entity via contrastive learning on noun phrases, and retrieves entities at inference to serve as hard prompts for a GPT-2 decoder that also uses CLIP soft prompts.
Key results
- Cross-domain MSCOCO to Flickr30k CIDEr 45.6 and Flickr30k to MSCOCO CIDEr 59.8, versus MeaCap's 43.8 and 56.4.
- In-domain on MSCOCO, CIDEr 96.0 vs MeaCap 95.4 and BLEU@4 27.3; on Flickr30k CIDEr 57.2 (MeaCap 59.4).
- NoCaps overall CIDEr 69.4 vs ViECap 66.2 among text-only methods; ablation shows K=5 representations per entity is best on Flickr30k (CIDEr 57.2 vs 55.7 for K=1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contrastive-trained multi-type entity representations for retrieval | MERCap · MSCOCO→Flickr30k | CIDEr: 43.8 -> 45.6 | Helped |
| Add contrastive-trained multi-type entity representations for retrieval | MERCap · Flickr30k→MSCOCO | CIDEr: 56.4 -> 59.8 | Helped |
| Add contrastive-trained multi-type entity representations for retrieval | MERCap · NoCaps val overall | CIDEr: 66.2 -> 69.4 | Helped |
| Add contrastive-trained multi-type entity representations for retrieval | MERCap · MSCOCO test | CIDEr: 95.4 -> 96.0 | Helped |
| Add contrastive-trained multi-type entity representations for retrieval | MERCap · Flickr30k test | CIDEr: 59.4 -> 57.2 | Hurt |
| Add retrieved entities to the decoder input (ablation) | MERCap · Flickr30k test | CIDEr: 13.8 -> 57.2 | Helped |
| Use multiple entity representations instead of one (K = 5) (ablation) | MERCap · Flickr30k test | CIDEr: 55.7 -> 57.2 | Helped |
| Set entity representation count to K = 3 instead of K = 5 (ablation) | MERCap · Flickr30k test | CIDEr: 57.2 -> 57.0 | Hurt |
| Set entity representation count to K = 7 instead of K = 5 (ablation) | MERCap · Flickr30k test | CIDEr: 57.2 -> 56.4 | Hurt |
| Add soft prompts (ablation) | MERCap · Flickr30k test | CIDEr: 18.3 -> 57.2 | Helped |
| Add neighbor texts to the soft prompts (ablation) | MERCap · Flickr30k test | CIDEr: 49.6 -> 57.2 | Helped |
| Use contrastive learning instead of CLIP entity embeddings (ablation) | MERCap · Flickr30k test | CIDEr: 55.0 -> 57.2 | Helped |
Surprising findings
- Higher numbers of entity representations (K=7) reduce performance relative to K=5, so more is not always better.
- Entity retrieval recall does not always correlate positively with caption quality metrics, according to the authors' exploration.
Limitations
- Performance on in-domain Flickr30k CIDEr trails MeaCap (57.2 vs 59.4), and some metrics are not best.
- Relationship between entity recall and caption quality is not consistently positive, which the authors leave unresolved.
- Entity set is built from nouns above a frequency threshold and relies on an external vocabulary for out-of-domain retrieval.
Experimental setup
Datasets: MSCOCO Flickr30k NoCaps VGOI vocabulary COCO vocabulary
Benchmarks: MSCOCO Flickr30k NoCaps
Metrics: BLEU@4 METEOR CIDEr SPICE R@5
Baselines: MAGIC CapDec DeCap Knight ViECap MeaCap OSCAR I-Tuning SmallCap ZeroCap
Builds on: CLIP ZeroCap ViECap CapDec K-Nearest-neighbor soft prompts (Knight)
Closest papers by meaning
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- MeaCap: Memory-Augmented Zero-shot Image Captioning (CVPR 2024 · similarity 0.87)
- ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning (AAAI 2025 · similarity 0.86)
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.86)
- Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions (CVPR 2024 · similarity 0.86)
- Image Captioning with Multi-Context Synthetic Data (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.