Research map / Vision-language and multimodal / Visual question answering
Multimodal question answering datasets
Creates benchmarks and models for multi-hop, video, and text-based multimodal question answering.
90 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Visual question answering, keyword cluster “answer · qa · video question”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
19 papers in 2023–24 and 32 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 17.1% to 18.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Stacked Attention Networks for Image Question Answering (CVPR 2016 · 2,094 citations)
- 2017 · TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (CVPR 2017 · 480 citations)
- 2018 · IQA: Visual Question Answering in Interactive Environments (CVPR 2018 · 376 citations)
- 2019 · GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (CVPR 2019 · 1,458 citations)
- 2020 · Reasoning with Heterogeneous Graph Alignment for Video Question Answering (AAAI 2020 · 188 citations)
- 2021 · MultiModalQA: complex question answering over text, tables and images (ICLR 2021 · 12 citations)
- 2022 · Invariant Grounding for Video Question Answering (CVPR 2022 · 118 citations)
- 2023 · SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images (AAAI 2023 · 56 citations)
- 2024 · Can I Trust Your Answer? Visually Grounded Video Question Answering (CVPR 2024 · 62 citations)
- 2025 · VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents (CVPR 2025 · 15 citations)
- 2026 · MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering (AAAI 2026 · 1 citation)
Earliest papers here
- Stacked Attention Networks for Image Question Answering (CVPR 2016 · 2,094 citations)
- Visual7W: Grounded Question Answering in Images (CVPR 2016 · 804 citations)
- Multimodal Residual Learning for Visual QA (NeurIPS 2016 · 235 citations)
What has been tried
74 papers here report 803 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 4%, hurt 23%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add Gaussian mask optimization and cross-modal self-supervision | NExT-GQA test · Acc@QA | +1.7 | Helped | Can I Trust Your Answer? Visually Grounded Video Question Answering (CVPR 2024 · 62 citations) |
| Add event parsing and reasoning to the no-stage JCEF baseline | NExT-QA val · accuracy | 66.7 -> 68.3 | Helped | MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · 43 citations) |
| Add QCD to a network without QCD or MCC | MUSIC-AVQA (All) · accuracy | 26.09 -> 71.94 | Helped | Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering (AAAI 2024 · 20 citations) |
| Remove RCR pre-training | SlideVQA single-pool · nDCG@5 | 77.3 -> 75.9 (−1.4) | Hurt | VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents (CVPR 2025 · 15 citations) |
| Fine-tune TimeChat on the collected instructions | MultiHop-EgoQA · mIoU | 3.6 -> 8.1 | Helped | Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos (AAAI 2025 · 14 citations) |
| add GSG to CRA | NextGQA test · Acc@GQA | 16.4 (↓1.8) -> 18.2 | Helped | Cross-modal Causal Relation Alignment for Video Question Grounding (CVPR 2025 · 13 citations) |
| use an LSTM instead of a Transformer text encoder | VTQA English test · EM | 55.45; 55.07; 54.07 -> 57.95 | Helped | VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · 12 citations) |
| Drop dependent questions whose parent answer is NO instead of counting them as NO | TIFA160 · Spearman's ρ / Kendall's τ | 0.571/0.458 -> 0.561/0.451 | No clear effect | Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation (ICLR 2024 · 8 citations) |
| Add Question-Aware Fusion to the uniform-sampling baseline | MUSIC-AVQA · accuracy | 75.04 -> 76.53 | Helped | Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · 7 citations) |
| Remove sparse sampling and vision refinement, using evenly sampled 16 frames and a 3-layer transformer | EgoTaskQA · accuracy | 29.10 -> 45.02 | Hurt | Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · 5 citations) |
| Increase perturbation range from 0.1-0.3 to 0.5-0.7 | Directional Guidance benchmark · Overall Accuracy | 0.24 -> 0.49 | Helped | Right this way: Can VLMs Guide Us to See More to Answer Questions? (NeurIPS 2024 · 5 citations) |
| permute subtitle features | TVQA subtitle-biased questions · accuracy (%) | 85.9 ± 0.0 -> 34.2 ± 3.4 (-51.7) | Hurt | Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models (AAAI 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 22 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MUSIC-AVQA · 2 of 22 papers
- GQA · 2 of 22 papers
- NExT-QA · 2 of 22 papers
- NExT-GQA · 2 of 22 papers
- DocVQA · 2 of 22 papers
- ChartQA · 2 of 22 papers
- ScanQA · 2 of 22 papers
- Charades · 2 of 22 papers
Baselines
- GPT-4o · 5 of 22 papers
- FrozenBiLM · 2 of 22 papers
- AVSD · 2 of 22 papers
- Pano-AVQA · 2 of 22 papers
- ST-AVQA · 2 of 22 papers
- PSTP-Net · 2 of 22 papers
- APL · 2 of 22 papers
- LLaVA-1.5 · 2 of 22 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 51 papers; from OpenAlex and the papers' affiliations).
- Nanyang Technological University · 4 papers
- Shanghai Jiao Tong University · 3 papers
- Tsinghua University · 3 papers
- Chinese Academy of Sciences · 2 papers
- University of Chinese Academy of Sciences · 2 papers
- Samsung · 2 papers
- NTT Corporation · 2 papers
- University of Copenhagen · 2 papers
- Huawei · 2 papers
- National University of Singapore · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Visual reasoning diagnostics (62 papers)
- Open-domain multimodal QA (73 papers)
Elsewhere in the topic
- Attention and prior-robust VQA (109 papers)
- Universal VQA models (100 papers)
- Vision-language model VQA evaluation (56 papers)
- Multimodal LLM question answering (56 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).