Research map / Vision-language and multimodal / Visual question answering
Open-domain multimodal QA
Answers questions over multiple evidence sources with co-attention, cross-lingual retrieval, and dialog.
73 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Visual question answering, keyword cluster “answer · qa · video question”
State History Benchmarks Tried Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
13 papers in 2023–24 and 13 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 11.7% to 7.5%.
No longer reported by papers here: WikiQA (MRR, last in 2020).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Dynamic Memory Networks for Visual and Textual Question Answering (ICML 2016 · 674 citations)
- 2017 · Dynamic Coattention Networks For Question Answering (ICLR 2017 · 502 citations)
- 2018 · R3: Reinforced Ranker-Reader for Open-Domain Question Answering (AAAI 2018 · 305 citations)
- 2019 · Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence (CVPR 2019 · 94 citations)
- 2020 · Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks (AAAI 2020 · 66 citations)
- 2021 · Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering (AAAI 2021 · 115 citations)
- 2022 · Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (NeurIPS 2022 · 215 citations)
- 2023 · Large Language Models Struggle to Learn Long-Tail Knowledge (ICML 2023 · 96 citations)
- 2024 · UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models (AAAI 2024 · 24 citations)
- 2025 · Audio Entailment: Assessing Deductive Reasoning for Audio Understanding (AAAI 2025 · 7 citations)
- 2026 · MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio (ICML 2026 · 0 citations)
Earliest papers here
- Dynamic Memory Networks for Visual and Textual Question Answering (ICML 2016 · 674 citations)
- Commonsense in Parts: Mining Part-Whole Relations from the Web and Image Tags (AAAI 2016 · 22 citations)
- Dynamic Coattention Networks For Question Answering (ICLR 2017 · 502 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
WikiQA · MRR (higher is better)
4 papers, 2018–2020 · not reported since 2020
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 74.3 | Improved Text Matching by Enhancing Mutual Information (AAAI 2018 · 12 citations) |
| 2020 | 82.22 | Hashing Based Answer Selection (AAAI 2020 · 8 citations) |
What has been tried
63 papers here report 638 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 8%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add a shared encoder | MS MARCO · R@1 | 37.89 -> 38.75 | Helped | UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models (AAAI 2024 · 24 citations) |
| Fine-tune on SPIQA QA data | SPIQA test-A · L3Score | 7.50 -> 43.90 | Helped | SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers (NeurIPS 2024 · 12 citations) |
| Caption audio before reasoning | CLE zero-shot · F1 | 0.4975 -> 0.5601 | Helped | Audio Entailment: Assessing Deductive Reasoning for Audio Understanding (AAAI 2025 · 7 citations) |
| use triplet-level graph features instead of node features | CSQA · accuracy | 73.1 -> 76.1 | Helped | Graph Reasoning Transformers for Knowledge-Aware Question Answering (AAAI 2024 · 4 citations) |
| use GPT-4V as sub-question solver instead of OmniSearch (Q) | Dyn-VQA · F1-Recall | 41.20 -> 43.15 | Helped | Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · 3 citations) |
| Use neighboring equations rather than all equations | cPAPERS-EQNS test · ROUGE-1 | 0.170 -> 0.190 | Helped | cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers (NeurIPS 2024 · 1 citation) |
| Add binary reward r_b | GuessWhat?! New Object, greedy, 5 rounds · success rate (%) | 60.09 -> 62.06 | Helped | Divide-and-Conquer: Tree-structured Strategy with Answer Distribution Estimator for Goal-Oriented Visual Dialogue (AAAI 2025 · 0 citations) |
| Replace vanilla query summaries with SR summaries generated using SLU labels | RETQA table retrieval · F1 | 91.42 -> 94.03 | Helped | RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector (AAAI 2025 · 0 citations) |
| Add row and column patches | MMTU · accuracy | Helped | Compositional Condition Question Answering in Tabular Understanding (ICML 2025) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 26 papers; from OpenAlex and the papers' affiliations).
- Carnegie Mellon University · 3 papers
- Peng Cheng Laboratory · 2 papers
- Renmin University of China · 2 papers
- Google · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Multimodal question answering datasets (90 papers)
- Visual reasoning diagnostics (62 papers)
Elsewhere in the topic
- Attention and prior-robust VQA (109 papers)
- Universal VQA models (100 papers)
- Vision-language model VQA evaluation (56 papers)
- Multimodal LLM question answering (56 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).