Research map / Vision-language and multimodal / Visual question answering
Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RepARe, a gradient-free, zero-shot framework where an LVLM extracts visual details (keywords, rationales, captions) to rephrase and augment underspecified VQA questions, selecting the candidate by answer confidence.
Problem
Zero-shot large vision-language models often answer underspecified VQA questions incorrectly due to missing visual information, implicit reasoning needs, or ambiguity; the input phrasing strongly affects performance.
Approach
Use the LVLM itself as captioner and rationale generator to extract salient visual details, fuse them into the question with the frozen LLM to create n candidate rephrasings, and select the candidate whose generated answer has the highest LVLM confidence, without gold answers or training.
Key results
- RepARe improves BLIP-2 zero-shot VQAv2 overall accuracy by 3.85 points (62.58 to 66.43 with Flan-T5-xl).
- On VizWiz, RepARe improves MiniGPT-4 Vicuna 13B by 7.94 points? Reported gains on VizWiz with MiniGPT-4 reach 7.94 points (Vicuna 7B: 29.87 to 37.81 is 7.94).
- Oracle selection with gold answers yields up to 14.41 point gains on A-OKVQA direct answer with MiniGPT-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rephrase and augment questions with visually grounded details | BLIP-2 Flan T5-xl · VQAv2 val | accuracy: 62.58 -> 66.43 | Helped |
| Rephrase and augment questions with visually grounded details | BLIP-2 Flan T5-xl · A-OKVQA Direct val | accuracy: 41.86 -> 44.87 | Helped |
| Rephrase and augment questions with visually grounded details | BLIP-2 Flan T5-xl · A-OKVQA MC val | accuracy: 73.89 -> 77.20 | Helped |
| Rephrase and augment questions with visually grounded details | BLIP-2 Flan T5-xl · VizWiz val | accuracy: 59.63 -> 62.38 | Helped |
| Add rationale-derived details (ablation) | RepARe with BLIP-2 · VQAv2 dev | accuracy: 64.57 -> 67.28 | Helped |
| Add caption-derived details (ablation) | RepARe with BLIP-2 · VQAv2 dev | accuracy: 66.04 -> 67.28 | Helped |
| Add question-entity details (ablation) | RepARe with BLIP-2 · VQAv2 dev | accuracy: 65.89 -> 67.28 | Helped |
| Exclude image embeddings during question fusion (ablation) | RepARe with BLIP-2 · VQAv2 dev | accuracy: 63.18 -> 67.28 | Helped |
| Select candidates using answer confidence rather than question likelihood (ablation) | RepARe with BLIP-2 · VQAv2 dev | accuracy: 65.47 -> 67.28 | Helped |
| Select from rephrased candidates instead of paraphrased candidates | BLIP-2 · VQAv2 val | accuracy: 62.91 -> 66.43 | Helped |
| Select from rephrased candidates instead of paraphrased candidates | BLIP-2 · A-OKVQA Direct val | accuracy: 40.23 -> 44.87 | Helped |
| Increase the number of candidates from n=2 to n=5 (ablation) | RepARe with BLIP-2 · VQAv2 and A-OKVQA | accuracy | Helped |
Surprising findings
- Paraphrasing questions and selecting by model confidence yields lower performance than original questions in 2 of 3 settings, while RepARe selection still improves.
- Including image embeddings in the sentence fusion step hurts performance (up to 3.1 point drop), so fusion uses only the frozen LLM.
Limitations
- Inference cost scales linearly with the number of question candidates.
- Veracity and utility of generated rationales are not verified.
- Selection via answer confidence degrades as candidate pool size grows (n=15).
Experimental setup
Datasets: VQAv2 A-OKVQA VizWiz
Benchmarks: VQAv2 A-OKVQA VizWiz
Metrics: VQA soft accuracy accuracy (multiple choice) average dependency distance idea density
Baselines: Paraphrase oracle (Pegasus paraphrasing) Paraphrase selection Original question zero-shot
Builds on: BLIP-2 MiniGPT-4 LLaVA-1.5
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.91)
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.91)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.91)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.91)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.90)
- From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.