Research map / Vision-language and multimodal / CLIP and zero-shot
MeaCap: Memory-Augmented Zero-shot Image Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeaCap, a memory-augmented zero-shot image captioning framework that retrieves CLIP-matched sentences from a text memory, filters them into key concepts, and guides CBART with a memory-augmented fusion score, in training-free and text-only-training variants.
Problem
Zero-shot image captioning without paired image-text data suffers from hallucination in training-free methods and loss of world knowledge/generalization in text-only-training methods.
Approach
Use the captioning text corpus as an external textual memory rather than to train the LM: retrieve image-related sentences with CLIP, parse them into subject-predicate-object concepts, merge and filter them into key concepts, and generate concept-centered captions with CBART guided by a fusion score combining LM fluency, CLIP image-text similarity, and Sentence-BERT memory-caption similarity.
Key results
- On MSCOCO zero-shot (CC3M transfer), MeaCapToT reaches CIDEr 48.3 and overall NoCaps CIDEr 45.1, versus 39.7 for DeCap.
- The training-free MeaCapTF gets MSCOCO CIDEr 42.5 and BLIP2-S 0.81, outperforming other training-free methods (ZeroCap, Tewel et al., ConZIC) on reference-based metrics.
- In-domain MSCOCO, MeaCapToT achieves CIDEr 84.8 versus 53.4 for CLIPRe and 55.4 for ZEROGEN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add memory-augmented key-concept retrieval and filtering (ablation) | MeaCapTF · MSCOCO | CIDEr: 12.7 -> 31.1 | Helped |
| Add image-text similarity guidance to retrieve-then-filter (ablation) | MeaCapTF · MSCOCO | CIDEr: 31.1 -> 38.6 | Helped |
| Add text-text similarity to the ReF and ITs configuration (ablation) | MeaCapTF · MSCOCO | CIDEr: 38.6 -> 42.5 | Helped |
| Add memory-augmented key-concept retrieval and filtering (ablation) | MeaCapToT · MSCOCO | CIDEr: 17.3 -> 37.1 | Helped |
| Add image-text similarity guidance to retrieve-then-filter (ablation) | MeaCapToT · MSCOCO | CIDEr: 37.1 -> 44.7 | Helped |
| Add text-text similarity to the ReF and ITs configuration (ablation) | MeaCapToT · MSCOCO | CIDEr: 44.7 -> 48.3 | Helped |
| Fine-tune CBART on CC3M while retaining the external memory | MeaCap · MSCOCO Karpathy-test, zero-shot | CIDEr: 42.5 -> 48.3 | Helped |
| Use retrieved key concepts and memory guidance with an CLIP-invert language decoder | MeaCapInvLM · MSCOCO, in-domain | CIDEr: 92.9 -> 95.4 | Helped |
Limitations
- Training-free variant has lower CLIP-S than prior training-free methods, which the authors attribute to CLIP-S favoring hallucination.
- Depends on a fixed set of hyperparameters (threshold tau, weights alpha, beta, gamma) chosen empirically.
Experimental setup
Datasets: CC3M MSCOCO Flickr30K NoCaps
Benchmarks: MSCOCO Karpathy test NoCaps validation Flickr30K Karpathy test
Metrics: BLEU-4 METEOR CIDEr SPICE CLIPScore BLIP2-Score
Baselines: ZeroCap Tewel et al. ConZIC CLIPRe DeCap MAGIC ZEROGEN CapDec ViECap
Builds on: CBART CLIP DeCap ZeroCap MAGIC
Closest papers by meaning
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- Zero-Shot Image Captioning with Multi-type Entity Representations (AAAI 2025 · similarity 0.87)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.87)
- A Simple Framework for Open-Vocabulary Zero-Shot Segmentation (ICLR 2025 · similarity 0.86)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.86)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.