Research map / Vision-language and multimodal / Visual question answering
Vision-language model VQA evaluation
Generates challenging benchmarks and zero-shot prompting for evaluating vision-language models on VQA.
56 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Visual question answering, keyword cluster “visual question · answering vqa · knowledge”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
13 papers in 2023–24 and 22 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 11.7% to 12.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · What Value Do Explicit High Level Concepts Have in Vision to Language Problems? (CVPR 2016 · 482 citations)
- 2017 · Modulating early visual processing by language (NeurIPS 2017 · 414 citations)
- 2019 · ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (NeurIPS 2019 · 1,665 citations)
- 2020 · Large-Scale Adversarial Training for Vision-and-Language Representation Learning (NeurIPS 2020 · 277 citations)
- 2021 · Counterfactual VQA: A Cause-Effect Look at Language Bias (CVPR 2021 · 420 citations)
- 2022 · Playing Lottery Tickets with Vision and Language (AAAI 2022 · 32 citations)
- 2023 · From Images to Textual Prompts: Zero-Shot Visual Question Answering With Frozen Large Language Models (CVPR 2023 · 138 citations)
- 2024 · Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering (CVPR 2024 · 20 citations)
- 2025 · VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis (AAAI 2025 · 30 citations)
- 2026 · STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes (AAAI 2026 · 5 citations)
Earliest papers here
- What Value Do Explicit High Level Concepts Have in Vision to Language Problems? (CVPR 2016 · 482 citations)
- Yin and Yang: Balancing and Answering Binary Visual Questions (CVPR 2016 · 320 citations)
- Modulating early visual processing by language (NeurIPS 2017 · 414 citations)
What has been tried
42 papers here report 467 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 7%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add VersaD RS pretraining | Scene classification average · accuracy | 73.22 -> 84.71 | Helped | VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis (AAAI 2025 · 30 citations) |
| replace demonstration images with mismatched images | VQAv2, VizWiz, and OK-VQA · accuracy | 45.97 -> 45.13; 27.00 -> 26.92; 36.13 -> 35.96; 49.94 -> 49.73; 27.21 -> 26.92; 36.68 -> 36.16 | No clear effect | How to Configure Good In-Context Sequence for Visual Question Answering (CVPR 2024 · 13 citations) |
| Add DSI only | VQA 2.0 val · overall accuracy | 67.27 -> 67.58 | Helped | Exploiting the Social-Like Prior in Transformer for Visual Reasoning (AAAI 2024 · 8 citations) |
| Train on simulated observations only instead of zero-shot evaluation | MetaVQA test set · accuracy | 0.592 -> 0.807 | Helped | Embodied Scene Understanding for Vision Language Models via MetaVQA (CVPR 2025 · 7 citations) |
| Add concept misunderstanding proposer | MMMU · average model performance | 43.8 -> 40.7 | Helped | Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation (CVPR 2025 · 6 citations) |
| Increase inner-monologue turns from 0 to 2 | ScienceQA test · accuracy | Helped | Tackling Vision Language Tasks through Learning Inner Monologues (AAAI 2024 · 4 citations) | |
| Adjust the answer threshold to enforce balanced predictions across paired image-question samples | NaturalBench · G-Acc | 40.5 -> 75.6 | Helped | NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · 4 citations) |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ | FF++ · F1 | 73.10 -> 76.83 (+3.73%) | Helped | A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · 4 citations) |
| Set optimization threshold to ±0.5 instead of ±1.0 | DVE generation optimization, Round3 · score | 0.9163 -> 0.9384 | Helped | Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization (AAAI 2025 · 3 citations) |
| fine-tune with 5 trajectories instead of no fine-tuning | UR5 cloth sweep · balanced accuracy | 49% -> 92% | Helped | Vision-Language Models for Robot Success Detection (AAAI 2024 · 2 citations) |
| Initialize module directions with learned directions rather than ground-truth module directions | Sine-wave regression · generalization performance | No clear effect | Breaking Neural Network Scaling Laws with Modularity (ICLR 2025 · 1 citation) | |
| Add rationale-derived details | VQAv2 dev · accuracy | 64.57 -> 67.28 | Helped | Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 12 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- VQAv2 · 2 of 12 papers
- A-OKVQA · 2 of 12 papers
- SEEDBench · 2 of 12 papers
- GQA · 2 of 12 papers
Baselines
- GPT-4o · 5 of 12 papers
- Qwen-VL-Chat · 3 of 12 papers
- InstructBLIP · 2 of 12 papers
- CogVLM · 2 of 12 papers
- LLaVA-1.5 · 2 of 12 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 35 papers; from OpenAlex and the papers' affiliations).
- Nanyang Technological University · 3 papers
- Google · 3 papers
- Harbin Institute of Technology · 2 papers
- The University of Sydney · 2 papers
- Stanford University · 2 papers
- University of California, Berkeley · 2 papers
- University of Washington · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Universal VQA models (100 papers)
- Attention and prior-robust VQA (109 papers)
- Multimodal LLM question answering (56 papers)
Elsewhere in the topic
- Multimodal question answering datasets (90 papers)
- Open-domain multimodal QA (73 papers)
- Visual reasoning diagnostics (62 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).