Research map / Vision-language and multimodal / Multimodal LLMs
Multimodal LLM benchmarks
Builds comprehensive and evolving benchmarks for large multimodal models.
252 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Multimodal LLMs, keyword cluster “vlms · visual reasoning · chain”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
20 papers in 2023–24 and 230 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 9.7% to 10.7%.
Open questions the data points to
- Papers report LISA-7B on ReasonSeg (gIoU) anywhere from 44.4 to 52.9: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Towards Building Large Scale Multimodal Domain-Aware Conversation Systems (AAAI 2018 · 111 citations)
- 2020 · V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices (AAAI 2020 · 24 citations)
- 2024 · MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (CVPR 2024 · 355 citations)
- 2025 · ChatGarment: Garment Estimation, Generation and Editing via Large Language Models (CVPR 2025 · 16 citations)
- 2026 · GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · 4 citations)
Earliest papers here
- Towards Building Large Scale Multimodal Domain-Aware Conversation Systems (AAAI 2018 · 111 citations)
- V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices (AAAI 2020 · 24 citations)
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (CVPR 2024 · 355 citations)
What has been tried
83 papers here report 740 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 5%, hurt 23%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add MUSE training data to LISA | ReasonSeg val · gIoU | 52.9 -> 58.0 | Helped | PixelLM: Pixel Reasoning with Large Multimodal Model (CVPR 2024 · 81 citations) |
| use multi-layer rather than single-layer CLIP features | Visual7W · accuracy | 82.44 -> 86.04 | Helped | ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · 73 citations) |
| Add totally right/wrong short-answer grading examples | MM-Vet evaluator human-score agreement · Average absolute difference | 0.0630 -> 0.0625 | Helped | MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · 59 citations) |
| Add Chain-of-Thought prompting | MATH-Vision · accuracy | 22.76 -> 23.98 | Helped | Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset (NeurIPS 2024 · 20 citations) |
| Add part-level garment descriptions to training data | CLoSE · CD | 4.06 -> 2.94 | Helped | ChatGarment: Garment Estimation, Generation and Editing via Large Language Models (CVPR 2025 · 16 citations) |
| add Mixed Long-CoT finetuning | GSM8K · score | 17.4 -> 65.2 | Helped | MMaDA: Multimodal Large Diffusion Language Models (NeurIPS 2025 · 16 citations) |
| Remove the image and provide only textual questions | ALM-bench, 50 sampled languages · performance | -27.3% | Hurt | All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages (CVPR 2025 · 9 citations) |
| grounding via textual choices (vs. image annotation or element attributes) | Cross-Task / Cross-Website / Cross-Domain 30-task subset · Step SR | 39.1 vs 20.3 (annotation), 16.1 (attributes) on Cross-Task | Helped | GPT-4V(ision) is a Generalist Web Agent, if Grounded (ICML 2024 · 9 citations) |
| Apply Stage 2 RefineBot after Stage 1 exam-style MCQ generation | MicroVQA · model accuracy | 79.7 -> 46.8; 75.6 -> 46.2; 82.2 -> 51.7; 77.3 -> 47.1; 81.6 -> 52.8; 80.1 -> 49.8; 79.1 -> 51.1 | Helped | MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research (CVPR 2025 · 7 citations) |
| Progressively train from 32K to 64K instead of direct 64K mixed training | MVBench · score | 71.3 -> 73.0 | Helped | Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models (NeurIPS 2025 · 7 citations) |
| Use expected score calculation instead of GPT-4o's single score | Flickr8k-Expert · Kendall τ_b | 59.118 -> 60.385 | Helped | G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o (AAAI 2025 · 4 citations) |
| Use a human-verified perception response instead of the model's perception response | ConvBench reasoning, pairwise grading · win rate | +11.21 average score | Helped | ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Ablation Capability for Large Vision-Language Models (NeurIPS 2024 · 4 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| LISA-7B | ReasonSeg | gIoU | 44.4 – 52.9 | 2 papers, e.g. NeurIPS 2025 and CVPR 2025 |
What reviewers will expect
Datasets and baselines named in the 230 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO · 10 of 230 papers
- RefCOCO · 7 of 230 papers
- PMC-VQA · 6 of 230 papers
- RefCOCOg · 5 of 230 papers
- ReasonSeg · 5 of 230 papers
- ADE20K · 5 of 230 papers
- ImageNet · 5 of 230 papers
- PubMedVision · 4 of 230 papers
Baselines
- GPT-4o · 62 of 230 papers
- Gemini-2.5-Pro · 27 of 230 papers
- Qwen2.5-VL-7B · 26 of 230 papers
- Qwen2.5-VL-72B · 23 of 230 papers
- GPT-5 · 18 of 230 papers
- Gemini-1.5-Pro · 15 of 230 papers
- Claude-3.5-Sonnet · 13 of 230 papers
- Gemini-2.5-Flash · 13 of 230 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 250 papers; from OpenAlex and the papers' affiliations).
- Peking University · 10 papers
- Shanghai AI Laboratory · 9 papers
- Zhejiang University · 8 papers
- Tsinghua University · 8 papers
- National University of Singapore · 7 papers
- Hong Kong University of Science and Technology · 7 papers
- The University of Hong Kong · 7 papers
- Stanford University · 7 papers
- University of Washington · 7 papers
- The Chinese University of Hong Kong · 7 papers
Use it
Download BibTeX of 22 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- VLM capability analysis (264 papers)
- Chain-of-thought multimodal reasoning (332 papers)
- Spatial reasoning in VLMs (207 papers)
- Reinforcement learning for visual reasoning (163 papers)
Elsewhere in the topic
- Instruction-tuned multimodal assistants (199 papers)
- Benchmarks and analysis of MLLMs (181 papers)
- Interleaved image-text generation with MLLMs (153 papers)
- Unified any-modality language models (152 papers)
- Attention-based visual token pruning (117 papers)
- Decoding-time hallucination mitigation (91 papers)
- Training-free token reduction for MLLMs (90 papers)
- Visual instruction tuning (68 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).