Research map / Vision-language and multimodal / Visual question answering
PromptCap: Prompt-Guided Image Captioning for VQA with GPT-3
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, Jiebo Luo
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use question-aware PromptCap captions instead of generic OFA captions (ablation) | GPT-3 in-context VQA · OK-VQA val | accuracy (%): 56.6 -> 60.4 | Improved |
| use PromptCap captions instead of generic OFA captions (ablation) | GPT-3 in-context VQA · A-OKVQA val (direct answer) | accuracy (%): 51.0 -> 56.3 | Improved |
| use PromptCap captions instead of generic OFA captions (ablation) | GPT-3 in-context VQA · VQAv2 val | accuracy (%): 64.9 -> 74.1 | Improved |
| use GPT-3 (175B) instead of Flan-T5-XXL (11B) as the language model (ablation) | PromptCap pipeline · OK-VQA val | accuracy (%): 42.0 -> 60.4 | Improved |
| use GPT-3 instead of Flan-T5-XXL (ablation) | PromptCap pipeline · VQAv2 val | accuracy (%): 70.9 -> 74.1 | Improved |
| increase in-context examples from 1 to 32 (ablation) | PromptCap + GPT-3 with random examples · OK-VQA val | accuracy (%): 46.5 -> 55.2 | Improved |
| retrieve in-context examples with CLIP instead of random (ablation) | PromptCap + GPT-3 · OK-VQA val, n=32 | accuracy (%): 55.2 -> 60.4 | Improved |
| use PromptCap captions instead of generic captions on out-of-domain WebQA (ablation) | GPT-3 8-shot · WebQA val | FL*Acc: 33.5 -> 34.5 | Improved |
| use PromptCap instead of OFA-Cap for COCO ground-truth caption similarity (ablation) | caption quality · COCO-GT captions CIDEr | CIDEr: 147.9 -> 150.1 | Improved |
Closest papers by meaning
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (AAAI 2022 · similarity 0.93)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.92)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.90)
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering (NeurIPS 2025 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.89)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
