Research map / Vision-language and multimodal / Multimodal LLMs
Benchmarks and analysis of MLLMs
Evaluates and probes multimodal LLMs, covering feature fusion, information flow, benchmarks, and jailbreak weaknesses.
181 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Multimodal LLMs, keyword cluster “mllm · instruction · llms”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 3 of 6 tracked benchmarks.
15 papers in 2023–24 and 165 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 7.2% to 7.7%.
Open questions the data points to
- MM-Vet (score): best went from 41.6 (up to 2024) to 45.2 in 2025 — Towards Semantic Equivalence of Tokenization in Multimodal LLM (ICLR 2025 · 2 citations)
- MME (score): best went from 1551 (up to 2024) to 1918 in 2025 — Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · 18 citations)
- POPE (score): best went from 87.2 (up to 2024) to 89.1 in 2025 — Towards Semantic Equivalence of Tokenization in Multimodal LLM (ICLR 2025 · 2 citations)
What looks settled
- GQA (score): best up to 2024 65.3, best in 2025–26 61.7 from 4 papers, at most 0.2% better, if at all.
- MMBench (accuracy): best up to 2024 79.4, best in 2025–26 68.1 from 3 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · Multimodal Summarization with Guidance of Multimodal Reference (AAAI 2020 · 99 citations)
- 2024 · SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · 145 citations)
- 2025 · MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · 119 citations)
- 2026 · Toward Multimodal Fake News Detection by Multi-perspective Rationale Generation and Verification (AAAI 2026 · 6 citations)
Earliest papers here
- Multimodal Summarization with Guidance of Multimodal Reference (AAAI 2020 · 99 citations)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · 145 citations)
- RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback (CVPR 2024 · 103 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MM-Vet · score (higher is better)
8 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 41.6 | Dense Connector for MLLMs (NeurIPS 2024 · 8 citations) |
| 2025 | 45.2 | Towards Semantic Equivalence of Tokenization in Multimodal LLM (ICLR 2025 · 2 citations) |
MME · score (higher is better)
8 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 1551 | DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · 2 citations) |
| 2025 | 1918 | Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · 18 citations) |
POPE · score (higher is better)
6 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 87.2 | DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · 2 citations) |
| 2025 | 89.1 | Towards Semantic Equivalence of Tokenization in Multimodal LLM (ICLR 2025 · 2 citations) |
GQA · score (higher is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 65.3 | DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · 2 citations) |
MMBench · accuracy (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 79.4 | Dense Connector for MLLMs (NeurIPS 2024 · 8 citations) |
SEED · score (higher is better)
4 papers, 2024–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 63.3 | DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · 2 citations) |
What has been tried
81 papers here report 881 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 6%, hurt 21%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Dense DPO (DDPO) with segment-level weighting instead of vanilla DPO | MHumanEval · All-type response-level hallucination rate | 57.5 -> 55.5 | Helped | RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback (CVPR 2024 · 103 citations) |
| Replace the grouping mechanism with a linear input layer | VQA v2 · accuracy | 66.7 -> 63.8 | Hurt | NExT-GPT: Any-to-Any Multimodal LLM (ICML 2024 · 94 citations) |
| Add text to the audio and video modalities | MER2023-SEMI · F1 | Helped | Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · 93 citations) | |
| Use full Q-Instruct instead of only Q-Pathway | Low-level image description (mix) · sum score | 3.74 -> 3.86 | Helped | Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models (CVPR 2024 · 82 citations) |
| Replace OpenAI CLIP with PathCLIP as PathAsst vision encoder | PathVQA · Closed accuracy | +1.2 | Helped | PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology (AAAI 2024 · 59 citations) |
| add look-back features to LTFM | MVTec AD, VisA, HeadCT, and BrainMRI · image-level AUROC | 92.8 -> 94.0; 90.5 -> 91.1; 96.6 -> 97.6; 93.5 -> 93.9 | Helped | Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models (CVPR 2025 · 38 citations) |
| Remove patch merging (equivalent to adding patch merging) | V* · accuracy | 58.3 -> 57.3 | Hurt | Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · 18 citations) |
| add LLM visual pretraining | CARLA Longest6 · DS | 47 -> 56 | Helped | DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · 17 citations) |
| block attention from last position to itself | GQA subset six tasks · answer probability | No clear effect | Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · 15 citations) | |
| Use external direct fusion rather than internal direct fusion with the same Triple layer set | Overall benchmark average · average score | 48.54 -> 49.18 | Helped | Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices (CVPR 2025 · 15 citations) |
| use LoRA rank 16 instead of rank 8 | UMRB-Partial · average performance | 59.45 -> 59.30 | Hurt | Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · 13 citations) |
| Decompose the raw query into 3 sub-queries rather than using a raw-query image | Hades dataset (GPT-4o) · ASR (%) | 3.20 -> 18.80 | Helped | Distraction is All You Need for Multimodal Large Language Model Jailbreaking (CVPR 2025 · 11 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| LLaVA | MMBench | accuracy | 59.1 – 64.3 | 2 papers, e.g. AAAI 2025 and NeurIPS 2024 |
What reviewers will expect
Datasets and baselines named in the 165 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO · 10 of 165 papers
- MSCOCO · 9 of 165 papers
- ScienceQA · 7 of 165 papers
- RefCOCO · 6 of 165 papers
- ImageNet · 6 of 165 papers
- MM-SafetyBench · 6 of 165 papers
- OKVQA · 5 of 165 papers
- MMMU · 5 of 165 papers
Baselines
- GPT-4o · 34 of 165 papers
- Gemini-2.5-Pro · 12 of 165 papers
- GPT-4V · 11 of 165 papers
- Qwen2.5-VL · 10 of 165 papers
- LLaVA-1.5 · 9 of 165 papers
- Claude-3.7-Sonnet · 8 of 165 papers
- MiniCPM-V-2.6 · 7 of 165 papers
- Gemini-1.5-Pro · 7 of 165 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 180 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 13 papers
- The Chinese University of Hong Kong · 12 papers
- Tsinghua University · 9 papers
- Shanghai Jiao Tong University · 8 papers
- Tencent · 8 papers
- Peking University · 7 papers
- Alibaba Group · 7 papers
- Nanjing University · 7 papers
- Shanghai AI Laboratory · 6 papers
- Wuhan University · 5 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Instruction-tuned multimodal assistants (199 papers)
- Unified any-modality language models (152 papers)
- Interleaved image-text generation with MLLMs (153 papers)
- Visual instruction tuning (68 papers)
Elsewhere in the topic
- Chain-of-thought multimodal reasoning (332 papers)
- VLM capability analysis (264 papers)
- Multimodal LLM benchmarks (252 papers)
- Spatial reasoning in VLMs (207 papers)
- Reinforcement learning for visual reasoning (163 papers)
- Attention-based visual token pruning (117 papers)
- Decoding-time hallucination mitigation (91 papers)
- Training-free token reduction for MLLMs (90 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).