Research map / Vision-language and multimodal / Visual question answering
Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts classification datasets (ImageNet, COCO, ActivityNet, OVAD) into open-ended VQA sub-benchmarks, uses WordNet/activity hierarchy follow-up questions, and selects metrics via a human study, finding GPT-4 and LLaMA-2 judges best.
Problem
Evaluating text-generative vision-language models on open-ended VQA is hard: free-form answers, ambiguous questions, and coarse aggregated metrics; existing automatic metrics penalize synonyms and long answers.
Approach
Turn classification datasets into oVQA sub-benchmarks for objects, activities and attributes using ground-truth crops, ask hierarchy-based follow-up questions when the first answer is too coarse, and compare text metrics against human judgments to choose an evaluation metric.
Key results
- Follow-up questions improve performance by 32% on average across tested models on ImageNet-oVQA.
- On the VQAv2 human study, GPT-4 10-shot reaches Pearson 0.972 with human judgments, versus 0.906 for Contains and 0.525 for EM.
- BLIP-2 models outperform other models on ActivityNet-oVQA and ImageNet-oVQA, while InstructBLIP and BLIP lead on OVAD-oVQA attribute classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ask a semantic-hierarchy follow-up question after the initial answer | VLMs on ImageNet-oVQA · ImageNet-oVQA cropped | ClipM@1: 57.10 -> 67.22 | Helped |
| Ask a semantic-hierarchy follow-up question after the initial answer | VLMs on ActivityNet-oVQA · ActivityNet-oVQA | ClipM@1: 55.08 -> 60.96 | Helped |
| Use VQA-optimized hyperparameters rather than default hyperparameters (ablation) | BLIP-2 OPT · ImageNet-oVQA cropped | ClipM@1: 57.10 -> 35.32 | Hurt |
| Use VQA-optimized hyperparameters rather than default hyperparameters (ablation) | BLIP-2 T5 · ImageNet-oVQA cropped | ClipM@1: 54.50 -> 43.02 | Hurt |
| Evaluate answers using GPT-4 10-shot instead of Cont | VQA answer evaluation · VQAv2 | Pearson correlation to human judgments: 0.906 -> 0.972 | Helped |
| Evaluate answers using GPT-4 10-shot instead of an NLP text metric | VQA answer evaluation · VQAv2 | Pearson correlation to human judgments: 0.223 -> 0.972 | Helped |
| Use Cont instead of EM for long-form LLaVA answers | LLaVA VQA evaluation · VQAv2 | Pearson correlation to human judgments: 0.000 -> 0.894 | Helped |
| Fine-tune on VQAv2 rather than use zero-shot models | VQA models · VQAv2 test-dev | EM: 62.80 -> 81.34 | Helped |
| Use generic pretrained models rather than VQA-finetuned models | VLMs on fine-grained object and activity classification · ImageNet-oVQA and ActivityNet-oVQA | ClipM@1 | Helped |
| Use VQA-tuned models rather than generic pretrained models | VLMs on attribute classification · OVAD-oVQA | Cont Syn | Helped |
| Use VQA-optimized generation settings that shorten answers (ablation) | BLIP-2 on ImageNet-oVQA · ImageNet-oVQA cropped | answer length: 7.94 -> 1.96 | Mixed |
Surprising findings
- EM gives 0% accuracy for LLaVA on VQAv2 even for reasonable answers, so Contains is preferred for instruction-tuned models.
- Simple text metrics such as Contains and BLEU-1 perform comparably to most NLP learned metrics (BLEURT, BEM, BertScore) on short VQA answers.
Limitations
- Follow-up threshold delta=0.37 chosen by qualitative evaluation.
- Long model outputs are truncated to about 45 words, affecting LLaVA outputs.
- Human study uses internal annotators and a relatively small set of 1000 VQAv2 questions.
Experimental setup
Datasets: ImageNet COCO ActivityNet OVAD VQAv2 GQA WordNet
Benchmarks: ImageNet-oVQA COCO-oVQA ActivityNet-oVQA OVAD-oVQA VQAv2 GQA
Metrics: ExactMatch Contains ClipMatch Pearson correlation with human judgment accuracy
Baselines: BLIP X2-VLM InstructBLIP LLaVA CLIP ViT-L-14 CLIP EVA01-g-14
Builds on: BLIP-2 EVA-CLIP LAVIS
Closest papers by meaning
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.91)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.91)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.91)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Improving Automatic VQA Evaluation Using Large Language Models (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.