Research map / Vision-language and multimodal / CLIP and zero-shot
ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViPCap, which turns retrieved captions into visual prompts via sampled Gaussian semantic features and patch-level retrieval, added to CLIP image features for a lightweight (14M trainable) captioner.
Problem
Lightweight retrieval-augmented image captioning uses retrieved text only as text prompts, so visual features rely solely on the CLIP vision encoder and detailed image descriptions are not reflected in the visual embedding space.
Approach
The ViP module maps retrieved-text CLIP embeddings to a learnable Gaussian, samples M semantic features, retrieves per-patch matches by cosine similarity, fuses them with image patch features through a single-layer feature fusion network, and adds the result to the visual features as a visual prompt before the GPT-2 cross-attention decoder.
Key results
- On COCO test, ViPCap (14M trainable) reaches 122.9 CIDEr and 37.7 B@4, exceeding SmallCap (119.7 CIDEr) and SmallCap Large (121.8 CIDEr).
- On NoCaps val, ViPCap achieves 93.8 in-domain CIDEr and 81.3 overall, surpassing lightweight baselines.
- Plugging the ViP module into CapDec and ViECap improves cross-domain COCO-to-NoCaps entire CIDEr by 1.9 and 1.0 points respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ViP to CapDec | CapDec · COCO test | CIDEr: 91.8 -> 94.2 | Helped |
| Add ViP to CapDec | CapDec · COCO ⇒ Flickr30k | CIDEr: 35.7 -> 35.8 | No clear effect |
| Add ViP to ViECap | ViECap · NoCaps val, Entire | CIDEr: 66.2 -> 67.2 | Helped |
| Add ViP to ViECap | ViECap · Flickr30k test | CIDEr: 47.9 -> 48.8 | Helped |
| Use α·ω_add instead of no added vector or sampled noise (ablation) | ViPCap · COCO test | CIDEr: 121.1 -> 122.9 | Helped |
| Add patch retrieval (ablation) | ViPCap · COCO test | CIDEr: 121.1 -> 121.9 | Helped |
| Add learnable vector ω (ablation) | ViPCap with patch retrieval · COCO test | CIDEr: 121.1 -> 121.9 | Helped |
| Add scale factor α to ω_add (ablation) | ViPCap with patch retrieval and ω · COCO test | CIDEr: 121.9 -> 122.9 | Helped |
| Use proposed FFN instead of sum, concat, MLP→Sum, or MLP→Concat (ablation) | ViPCap · COCO test | CIDEr: 119.1; 119.8; 119.8; 119.8 -> 122.9 | Helped |
| Set M=200 rather than other tested sample counts (ablation) | ViPCap · COCO test | CIDEr: 121.4; 121.1; 121.5; 121.4; 121.4; 121.1 -> 122.9 | Helped |
| Add ViP using the prompt “This image shows” | ViPCap · COCO val | CIDEr: 111.1 -> 116.0 | Helped |
| Add ViP using retrieval prompt | ViPCap · COCO val | CIDEr: 117.3 -> 119.9 | Helped |
Surprising findings
- Adding the ViP module to CapDec on COCO-to-Flickr30k cross-domain setting decreased B@4 by 1.6 points.
Limitations
- Retrieval limited to 3 captions due to CLIP's 77-token context length.
- Performance gains degrade for large M (M>200) as local representations become scattered.
- Cannot be applied to MAPL due to limited access to its training code, so evaluation there is limited.
Experimental setup
Datasets: COCO Flickr30k NoCaps
Benchmarks: COCO Flickr30k NoCaps
Metrics: BLEU@4 METEOR CIDEr SPICE
Baselines: SmallCap CapDec ViECap EVCap MAPL ClipCap I-Tuning CaMEL OSCAR BLIP-2
Builds on: SmallCap CapDec ViECap CLIP GPT-2
Closest papers by meaning
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.88)
- Sentence-level Prompts Benefit Composed Image Retrieval (ICLR 2024 · similarity 0.87)
- MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suites (ICLR 2026 · similarity 0.86)
- Zero-Shot Image Captioning with Multi-type Entity Representations (AAAI 2025 · similarity 0.86)
- Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models (ICLR 2025 · similarity 0.86)
- EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.