Research map / Vision-language and multimodal / Visual question answering
NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NaturalBench, a semi-automatically collected, vision-centric VQA benchmark of 10,000 human-verified paired-image questions that exposes large gaps between VLMs and humans and shows blind-model and answer-bias shortcuts are blocked.
Problem
Current VQA benchmarks can be solved by blind language models exploiting language priors and answer imbalances, so they overstate VLM visual reasoning; models struggle on natural image-question pairs that humans find easy.
Approach
Mine confounding image-text pairs that CLIP/BLIP-2 mismatch, use ChatGPT/GPT-4 to write questions with opposite answers for each image, and pair each question with two images (and each image with two questions) so that constant answers fail; add multi-tag skill annotation and debiased thresholding for evaluation.
Key results
- Across 55 VLMs, the best closed-source model OpenAI o3 reaches 55.1 G-Acc versus about 92.1 for humans; GPT-4o gets 39.6 G-Acc, and most open models are 5-20 points above chance.
- Debiasing the decision threshold raises GPT-4o G-Acc from 40.5 to 75.6 and LLaVA-1.5 from 14.4 to 49.7 (original vs debiased).
- A blind GPT-3.5 finetuned on previous benchmarks such as MMStar and MME beats chance, while on NaturalBench it fails to exceed chance performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adjust the answer threshold to enforce balanced predictions across paired image-question samples (ablation) | GPT-4o · NaturalBench | G-Acc: 40.5 -> 75.6 | Helped |
| Adjust the answer threshold to enforce balanced predictions across paired image-question samples (ablation) | LLaVA-1.5, Vicuna-13B · NaturalBench | G-Acc: 14.4 -> 49.7 | Helped |
| Select confounding image-text pairs with discriminative VLMs rather than random pairs (ablation) | Flickr VQA sample collection · Flickr-Adversarial vs. Flickr-Random | G-Acc: 72.5 -> 38.3 | Mixed |
| Fine-tune LLaVA-1.5 on NaturalBench training images | LLaVA-1.5 · NaturalBench English test split | G-Acc: 9% | Helped |
| Fine-tune GPT-4o with vision on NaturalBench | GPT-4o · NaturalBench English test split | G-Acc: 10% | Helped |
Surprising findings
- Blind GPT-3.5 finetuned on answer-only data exploits language priors on previous VQA benchmarks, sometimes matching LLaVA-1.5 finetuned with images.
- Models choose the same answer regardless of the image, so Q-Acc is consistently lower than I-Acc.
Limitations
- Samples may inherit biases from web-scraped datasets and foundation models, so human verification is needed.
- Analysis focuses on individual skill tags and not on combinations of skills.
- Training-data scaling is left to future work; the benchmark is evaluation-focused.
Experimental setup
Datasets: Flickr30K DOCCI XM3600 MME MMBench MMMU MMStar VQAv1 VQAv2 GQA
Benchmarks: NaturalBench NaturalBench-Retrieval MME MMBench MMMU MMStar Winoground
Metrics: accuracy Q-Acc I-Acc G-Acc debiased G-Acc text score image score group score
Baselines: GPT-4o GPT-4V Gemini 2.5 Pro OpenAI o3 LLaVA-1.5 Qwen2-VL InternVL2 LLaVA-OneVision Llama3.2-Vision Molmo Cambrian-1 BLIP-3
Builds on: CLIP Winoground VQAScore VQAv2
Closest papers by meaning
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.91)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.91)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.90)
- Vision-Language Models Do Not Understand Negation (CVPR 2025 · similarity 0.90)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.90)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.