Research map / Vision-language and multimodal / Visual question answering
Multimodal LLM question answering
Augments multimodal LLMs for knowledge-based and text-rich VQA and evaluates them dynamically.
56 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Visual question answering, keyword cluster “visual question · answering vqa · knowledge”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Few recent reports None of the tracked benchmarks has a report from 2025–26.
12 papers in 2023–24 and 35 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 10.8% to 20.2%.
No longer reported by papers here: OK-VQA (accuracy (%), last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Dual Attention Networks for Multimodal Reasoning and Matching (CVPR 2017 · 753 citations)
- 2018 · Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (CVPR 2018 · 420 citations)
- 2019 · BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection (AAAI 2019 · 227 citations)
- 2020 · Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text (CVPR 2020 · 123 citations)
- 2022 · Dual-Key Multimodal Backdoors for Visual Question Answering (CVPR 2022 · 35 citations)
- 2023 · REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory (CVPR 2023 · 85 citations)
- 2024 · Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models (CVPR 2024 · 120 citations)
- 2025 · Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · 15 citations)
- 2026 · RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understanding (AAAI 2026 · 2 citations)
Earliest papers here
- Dual Attention Networks for Multimodal Reasoning and Matching (CVPR 2017 · 753 citations)
- Hadamard Product for Low-rank Bilinear Pooling (ICLR 2017 · 180 citations)
- Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (CVPR 2018 · 420 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
OK-VQA · accuracy (%) (higher is better)
4 papers, 2023–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 59.1 | REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory (CVPR 2023 · 85 citations) |
What has been tried
43 papers here report 494 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 4%, hurt 21%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace conventional positional interpolation with patch-based high-resolution processing | DeepForm · score | 11.4 -> 40.6 | Helped | Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models (CVPR 2024 · 120 citations) |
| Add projected image patch embeddings | TextVQA · accuracy (%) | 37.58 -> 58.8 | Helped | BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · 94 citations) |
| Add a numerical tokenizer instead of using a pretrained text tokenizer | ETH/UCY AVG · ADE/FDE | 0.65/1.28 -> 0.48/0.88 | Helped | Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction (CVPR 2024 · 47 citations) |
| Use two-stage, two-model training rather than only the in-article reflective model | Encyclopedic-VQA test, single-hop · VQA accuracy | 21.1 -> 35.5 | Helped | Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · 15 citations) |
| add SCF-generated interpretable captions to binary judgment targets | Protocol 2 average over 11 datasets · HTER(%) | 14.47 -> 11.30 | Helped | Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models (AAAI 2025 · 14 citations) |
| add the Multi-modal Input Aligner | MIMOSeg Perspective I · mIoU | 0.607 -> 0.639 | Helped | MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · 10 citations) |
| Add the JTD-UAV pipeline (tracking, feature compression, LLM with joint training) versus plain Qwen2-VL trained on same data | TDUAV subsets 1-3 · BLEU-4, SPICE, ACC | BLEU-4 9.98 -> 12.75 (Subset 1); SPICE 0.2812 -> 0.3264; ACC 83.12 -> 92.52 | Helped | JTD-UAV: MLLM-Enhanced Joint Tracking and Description Framework for Anti-UAV Systems (CVPR 2025 · 6 citations) |
| Remove the causality filter | 10 benchmarks · accuracy | Mixed | Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models (CVPR 2024 · 6 citations) | |
| add the aspect validation query | Twitter2015 · Micro-F1 | 87.4 -> 88.0 | Helped | DEQA: Descriptions Enhanced Question-Answering Framework for Multimodal Aspect-Based Sentiment Analysis (AAAI 2025 · 5 citations) |
| Reduce I3D sliding-window stride from 16 to 4 | SocialGesture · Avg mAP | 5.94 -> 19.19 | Helped | SocialGesture: Delving into Multi-person Gesture Understanding (CVPR 2025 · 3 citations) |
| add correctness scoring | MMstar · Pearson | 0.903 -> 0.944 | Helped | UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · 3 citations) |
| Add contrastive loss | COCO val grounding · Binding Hits | 10.11 -> 56.3 | Helped | CTRL-O: Language-Controllable Object-Centric Visual Representation Learning (CVPR 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 24 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO · 5 of 24 papers
- TextVQA · 4 of 24 papers
- VizWiz · 4 of 24 papers
- VQAv2 · 3 of 24 papers
- MMStar · 3 of 24 papers
- ScienceQA · 3 of 24 papers
- DocVQA · 3 of 24 papers
- GQA · 3 of 24 papers
Baselines
- GPT-4o · 6 of 24 papers
- InternVL2-8B · 4 of 24 papers
- GPT-4V · 3 of 24 papers
- CLIP · 3 of 24 papers
- Qwen2-VL-7B-Instruct · 3 of 24 papers
- BLIP-2 · 2 of 24 papers
- InstructBLIP · 2 of 24 papers
- RadFM · 2 of 24 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 47 papers; from OpenAlex and the papers' affiliations).
- Chinese Academy of Sciences · 3 papers
- Peking University · 3 papers
- Baidu · 2 papers
- Google · 2 papers
- Alibaba Group · 2 papers
- Shanghai AI Laboratory · 2 papers
- University of Illinois Chicago · 2 papers
- University of Illinois at Chicago · 2 papers
- Zhejiang University · 2 papers
Use it
Download BibTeX of 30 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Universal VQA models (100 papers)
- Attention and prior-robust VQA (109 papers)
- Vision-language model VQA evaluation (56 papers)
Elsewhere in the topic
- Multimodal question answering datasets (90 papers)
- Open-domain multimodal QA (73 papers)
- Visual reasoning diagnostics (62 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).