Research map / Large language models / Retrieval and knowledge
Improving Automatic VQA Evaluation Using Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAVE, an LLM-based VQA metric that frames answer evaluation as a few-shot answer-rating task with rationales, and shows higher human-correlation than VQA Accuracy and soft/embedding baselines.
Problem
VQA Accuracy relies on exact string matching against human references, which underrates correct open-ended answers from generative VQA models evaluated in OOD settings.
Approach
Use an instruction-tuned LLM with in-context demonstrations to rate a candidate answer (1-3 scale) against multiple reference answers, generating a rationale before the rating, and map the rating to [0,1].
Key results
- LAVE with GPT-3.5 reaches overall Spearman 68.91 versus 60.13 for VQA Accuracy and 63.91 for soft VQA Accuracy.
- LAVE with Flan-T5-XXL (64.99 overall Spearman) and Vicuna (64.05) also outperform all baselines on average.
- LAVE recovers most failure cases: 16.33% of test questions where humans say correct but VQA Accuracy is below 0.5 get a LAVE score of 1.0 in the analysis setting, per the text's framing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LAVE with GPT-3.5 instead of the strongest baseline metric | VQA evaluation · All VQA model-dataset pairs | Spearman correlation: 63.91 -> 68.91 | Helped |
| use LAVE with Flan-T5 instead of VQA Accuracy | VQA evaluation · All VQA model-dataset pairs | Kendall correlation: 55.75 -> 62.14 | Helped |
| use LAVE with GPT-3.5 instead of CIDEr | VQA evaluation · BLIP-2 on OK-VQA | Spearman correlation: 53.23 -> 61.47 | Helped |
| use LAVE with GPT-3.5 instead of the strongest baseline | VQA evaluation · BLIP-VG on VQAv2 | Spearman correlation: 85.07 -> 83.63 | Hurt |
| use LAVE with GPT-3.5 instead of the strongest baseline | VQA evaluation · BLIP-VQA on OK-VQA | Spearman correlation: 70.25 -> 67.57 | Hurt |
| add demonstrations, using the full set rather than 1 demonstration (ablation) | LAVE Flan-T5 · Ablation validation settings | Spearman correlation: 59.07 -> 66.74 | Helped |
| increase demonstrations from 4 to the full set (ablation) | LAVE Flan-T5 · Ablation validation settings | Spearman correlation: 65.40 -> 66.74 | Helped |
| add rationale generation before rating (ablation) | LAVE Flan-T5 · BLIP-2 on VQAv2 and VG-QA | Spearman correlation: 58.67 -> 67.50; 63.87 -> 61.57 | Mixed |
| filter low-frequency reference answers (ablation) | LAVE Flan-T5 · Ablation validation settings | Spearman correlation: 64.89 -> 66.74 | Helped |
| add an image caption to the prompt (ablation) | LAVE Flan-T5 · Ablation validation settings | Spearman correlation: 66.74 -> 66.94 | No clear effect |
| use output-token probabilities to normalize and combine ratings (ablation) | LAVE · VQA evaluation | correlation with human judgment | No clear effect |
Surprising findings
- On binary (yes/no) questions, LAVE correlates worse than VQA Accuracy (GPT-3.5 Spearman 88.40 vs 91.37), and 38.42% of binary questions have contradictory yes and no references.
- BLIP answers on VQAv2 and OK-VQA show VQA Accuracy correlating better with humans than LAVE, reversing the general trend.
Limitations
- LLM-based metric is slow because rationales are generated autoregressively.
- GPT-3.5 variant requires a paid API.
- Weaker on binary questions and ambiguous questions with contradictory references; LLMs sometimes hallucinate judgments.
Experimental setup
Datasets: VQAv2 VG-QA OK-VQA
Benchmarks: VQAv2 VG-QA OK-VQA
Metrics: Spearman correlation Kendall correlation
Baselines: VQA Accuracy Soft VQA Accuracy METEOR CIDEr BERTScore S-BERTScore
Builds on: VQA Accuracy Super-NaturalInstructions BERTScore Sentence-BERT
Closest papers by meaning
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.90)
- Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation (CVPR 2025 · similarity 0.90)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.89)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.89)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.