Research map / Vision-language and multimodal / CLIP and zero-shot
EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVCAP, a retrieval-augmented LLM image captioner that retrieves object names from an expandable visual-name memory (LVIS + Stable Diffusion images) and fuses them via an attentive Q-Former, training only 3.97M parameters on COCO.
Problem
LLM-based image captioners struggle to describe novel or commonsense-violating objects and cannot cheaply keep object knowledge up to date, while retrieval-augmented captioners store redundant caption text and need many trainable parameters.
Approach
Replace caption-text retrieval with a key-value memory whose keys are visual embeddings of object images and whose values are object names; retrieved names are filtered by an attentive fusion Q-Former with learnable object-name query tokens and fed with visual features into a frozen Vicuna-13B, so the memory can be expanded without retraining.
Key results
- With 3.97M trainable parameters trained only on COCO, EVCAP reaches 140.1 CIDEr on COCO test, 119.3 overall on NoCaps val, and 84.4 CIDEr on Flickr30k test.
- EVCAP outperforms lightweight MiniGPT4 (3.94M params) by 10.5 CIDEr on COCO and NoCaps overall and 6.0 on Flickr30k.
- Expanding memory with 2396 WHOOPS objects raises WHOOPS CIDEr from 85.3 to 86.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attentive fusion with retrieved names from external visual-name memory (ablation) | EVCAP, replacing the ViT + Q-Former baseline · COCO test | CIDEr: 134.4 -> 140.1 | Helped |
| add attentive fusion with retrieved names from external visual-name memory (ablation) | EVCAP, replacing the ViT + Q-Former baseline · NoCaps val | CIDEr: 108.8 -> 119.3 | Helped |
| add attentive fusion with retrieved names from external visual-name memory (ablation) | EVCAP, replacing the ViT + Q-Former baseline · Flickr30k test | CIDEr: 76.8 -> 84.4 | Helped |
| add trainable image query tokens without attentive fusion (ablation) | EVCAP baseline · COCO test | CIDEr: 134.4 -> 134.1 | No clear effect |
| add WHOOPS object images and names to the external memory (ablation) | EVCAP · WHOOPS | CIDEr: 85.3 -> 86.3 | Helped |
| remove 30%, 60%, or 90% of LVIS memory data (ablation) | EVCAP · NoCaps val overall | CIDEr: 119.3 -> 118.8; 119.3 -> 119.0; 119.3 -> 118.3 | Hurt |
| remove 30%, 60%, or 90% of LVIS memory data (ablation) | EVCAP · Flickr30k test | CIDEr: 84.4 -> 85.0; 84.4 -> 85.1; 84.4 -> 83.6 | Mixed |
| add WHOOPS knowledge to LVIS external memory (ablation) | EVCAP · NoCaps val overall | CIDEr: 119.3 -> 119.0 | No clear effect |
| add WHOOPS knowledge to LVIS external memory (ablation) | EVCAP · Flickr30k test | CIDEr: 84.4 -> 84.9 | Helped |
Surprising findings
- With zero retrieved object names the model performs worst on NoCaps, but gains from retrieval come from the mechanism rather than the customized Q-Former alone.
- Specialist SOTAs with billions of parameters still outperform EVCAP by nearly 10 CIDEr on all benchmarks.
Limitations
- Cannot retrieve all objects in an image due to memory coverage limits, giving incomplete captions.
- Focus on object representation neglects other caption elements; limited style diversity as with other COCO-trained models.
- Performance still trails specialist billion-scale models by nearly 10 CIDEr.
Experimental setup
Datasets: COCO LVIS WHOOPS NoCaps Flickr30k
Benchmarks: COCO Karpathy test NoCaps validation Flickr30k test WHOOPS
Metrics: BLEU@4 METEOR CIDEr SPICE
Baselines: SmallCap MiniGPT-4 BLIP-2 ViECap ClipCap InstructBLIP REVEAL RCA-NOC
Builds on: BLIP-2 SmallCap REVEAL MiniGPT-4
Closest papers by meaning
- FlexCap: Describe Anything in Images in Controllable Detail (NeurIPS 2024 · similarity 0.88)
- EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models (AAAI 2025 · similarity 0.88)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models (CVPR 2024 · similarity 0.87)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.87)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.