Research map / Vision-language and multimodal / Visual question answering
Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QACap, a question-aware captioner (fine-tuned BLIP) that generates question-relevant captions for an off-the-shelf LLM, plus a GPT-4-built question-aware caption dataset, reaching SOTA on OK-VQA and A-OKVQA.
Problem
Knowledge-based VQA with caption-then-LLM pipelines relies on generic captions that may not contain the details needed for a specific question, and existing captioners (e.g., PromptCap) tend to answer the question directly rather than supply context.
Approach
A question-aware captioner that fuses question and image features (question-aware feature distillation) and aligns question-to-image attention with caption-generation attention via a JS-divergence loss (QARCL), trained on a GPT-4-refined dataset that avoids answer leakage and selects captions by answer helpfulness and CLIP truthness.
Key results
- QACap + Claude 3.5 reaches 68.2% accuracy on OK-VQA validation, with no external knowledge base and zero in-context shots.
- On A-OKVQA validation, QACap + Claude 3.5 gets 66.3% direct answer, 76.7% multiple choice, and 71.5% average, exceeding PromptCap + GPT-3 (64.8% average).
- Training on the QACap dataset adds 2.9 points on OK-VQA and 5.0 on A-OKVQA DA over PromptCap data alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use question-aware captions with QACap instead of the strongest listed baseline | KBVQA with Claude 3.5 · OKVQA val | accuracy: 66.8 -> 68.2 | Helped |
| Use QACap captions instead of PromptCap captions | A-OKVQA · A-OKVQA val, Direct Answer | accuracy: 56.3 -> 66.3 | Helped |
| Use QACap captions instead of PromptCap captions | A-OKVQA · A-OKVQA val, Multiple Choice | accuracy: 73.2 -> 76.7 | Helped |
| Add QAFD to the baseline (ablation) | QACap · OKVQA val | accuracy: 61.5 -> 62.9 | Helped |
| Add QAFD to the baseline (ablation) | QACap · A-OKVQA val, Direct Answer | accuracy: 52.4 -> 53.8 | Helped |
| Add QAFD to the baseline (ablation) | QACap · A-OKVQA val, Multiple Choice | accuracy: 67.7 -> 65.6 | Hurt |
| Add QARCL without PET (ablation) | QACap with QAFD · OKVQA val | accuracy: 62.9 -> 60.9 | Hurt |
| Add PET without QARCL (ablation) | QACap with QAFD · OKVQA val | accuracy: 62.9 -> 66.1 | Helped |
| Add QARCL when PET is enabled (ablation) | QACap with QAFD and PET · OKVQA val | accuracy: 66.1 -> 68.2 | Helped |
| Add the QACap training dataset to PromptCap training data (ablation) | QACap with Claude 3.5 · OKVQA val | accuracy: 65.3 -> 68.2 (+2.9) | Helped |
| Use the QACap dataset instead of a size-matched PromptCap dataset (ablation) | QACap · A-OKVQA val, Direct Answer | accuracy: 56.7 -> 61.5 (+4.8) | Helped |
| Use question-aware captions rather than general captions (ablation) | QACap · A-OKVQA Direct Answer | accuracy | Helped |
Surprising findings
- QARCL alone lowers accuracy (configuration c underperforms b), and only helps when combined with PET.
Limitations
- Multiple-choice gains from question-aware captions are minor because answer choices let the LLM guess without detailed context.
- Results depend on the chosen LLM, with performance varying across OK-VQA and A-OKVQA and across models.
Experimental setup
Datasets: OK-VQA A-OKVQA FVQA PromptCap dataset MSCOCO captions
Benchmarks: OK-VQA A-OKVQA
Metrics: accuracy soft VQA accuracy
Baselines: BLIP-2 Flamingo CogVLM CogAgent InstructBLIP PaLI-X-VPD KCKR PICa Img2LLM KAT REVIVE PromptCap
Builds on: BLIP PromptCap Img2LLM Prophet
Closest papers by meaning
- Prompting Large Language Models With Answer Heuristics for Knowledge-Based Visual Question Answering (CVPR 2023 · similarity 0.94)
- TOA: Task-oriented Active VQA (NeurIPS 2023 · similarity 0.93)
- StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answering (CVPR 2026 · similarity 0.92)
- REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (NeurIPS 2022 · similarity 0.92)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.91)
- BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.