Research map / Vision-language and multimodal / Visual question answering
Visual reasoning diagnostics
Diagnoses shortcuts and compositional reasoning in visual reasoning models using synthetic benchmarks.
62 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Visual question answering, keyword cluster “answer · qa · video question”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 16 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 18.9% to 9.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (CVPR 2017 · 1,831 citations)
- 2018 · FiLM: Visual Reasoning with a General Conditioning Layer (AAAI 2018 · 2,107 citations)
- 2019 · From Recognition to Cognition: Visual Commonsense Reasoning (CVPR 2019 · 785 citations)
- 2020 · 12-in-1: Multi-Task Vision and Language Representation Learning (CVPR 2020 · 415 citations)
- 2021 · Unifying Vision-and-Language Tasks via Text Generation (ICML 2021 · 188 citations)
- 2022 · Flamingo: a Visual Language Model for Few-Shot Learning (NeurIPS 2022 · 1,238 citations)
- 2023 · Visual Programming: Compositional Visual Reasoning Without Training (CVPR 2023 · 202 citations)
- 2024 · Generative Multimodal Models are In-Context Learners (CVPR 2024 · 97 citations)
- 2025 · Evaluating Image Hallucination in Text-to-Image Generation with Question-Answering (AAAI 2025 · 8 citations)
- 2026 · ImageSet2Text: Describing Sets of Images Through Text (AAAI 2026 · 1 citation)
Earliest papers here
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (CVPR 2017 · 1,831 citations)
- Visual Dialog (CVPR 2017 · 486 citations)
- GuessWhat?! Visual Object Discovery Through Multi-Modal Dialogue (CVPR 2017 · 407 citations)
What has been tried
52 papers here report 589 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 6%, hurt 26%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Apply instruction tuning and additional instruction-aware tokens | EgoThink · Average score | 49.6 -> 51.5 | Helped | EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · 26 citations) |
| add anchor-based situation estimation | SQA3D · Acc@1.0m localization | 29.5 -> 58.8 | Helped | Situational Awareness Matters in 3D Vision Language Reasoning (CVPR 2024 · 15 citations) |
| increase semantic-tag training data from 1M to 5M images | COCO · mAP | 67.8 -> 68.6 | Helped | The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World (ICLR 2024 · 13 citations) |
| Use the strict scoring rule that marks a prompt incorrect if any of its five QA answers is wrong | I-HallA v1.0 across science and history · I-HallA score | See I-HallA Score^\dagger columns in the benchmark results table | Hurt | Evaluating Image Hallucination in Text-to-Image Generation with Question-Answering (AAAI 2025 · 8 citations) |
| add human-written corrections as in-context examples for hard question types | GQA · validation accuracy | Helped | Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement (CVPR 2024 · 7 citations) | |
| Use visual prompting QA with Q-A-Q-A encoding rather than Painter's Q-Q-A-A encoding | Poisson Noise · PSNR | 24.63 -> 27.29 | Helped | Unifying Image Processing as Visual Prompting Question Answering (ICML 2024 · 6 citations) |
| Add latent space translation (LST) | VQAI test set · Sim_Avg (CLIP) | 0.8414 -> 0.8589 | Helped | Image Content Generation with Causal Reasoning (AAAI 2024 · 5 citations) |
| jointly train on single-image and temporal tasks rather than temporal tasks only | fMoW RGB · accuracy | 73.5 -> 75.1 | Helped | TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data (ICLR 2025 · 4 citations) |
| add object names to bounding-box coordinate supervision | GQA testdev · Accuracy | 59.97 -> 59.40 | Hurt | Detection-Based Intermediate Supervision for Visual Question Answering (AAAI 2024 · 4 citations) |
| Use three vision encoders instead of CLIP alone | DocHaystack-1000 · Recall@1 | 23.9 -> 42.2 | Helped | Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · 2 citations) |
| Use only general questions rather than both general and label-specific questions | CIFAR-10 poisoned sample detection · Detection performance | Hurt | VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models (ICLR 2024 · 2 citations) | |
| Add zero-shot chain-of-thought prompting | Fish-Prompting species classification · accuracy | 79.00 -> 86.00 | Helped | VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images (NeurIPS 2024 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- DocVQA · 2 of 10 papers
- GQA · 2 of 10 papers
Baselines
- GPT-4o · 4 of 10 papers
- CLIP · 2 of 10 papers
- Qwen2-VL · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 37 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Tencent · 3 papers
- Huazhong University of Science and Technology · 2 papers
- Peking University · 2 papers
- A*STAR · 2 papers
- University of Pennsylvania · 2 papers
- The Chinese University of Hong Kong · 2 papers
- NVIDIA · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Multimodal question answering datasets (90 papers)
- Open-domain multimodal QA (73 papers)
Elsewhere in the topic
- Attention and prior-robust VQA (109 papers)
- Universal VQA models (100 papers)
- Vision-language model VQA evaluation (56 papers)
- Multimodal LLM question answering (56 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).