Research map / Vision-language and multimodal / Multimodal LLMs
VLM capability analysis
Probes and refines vision-language models, assessing priors, reasoning, and design choices.
264 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Multimodal LLMs, keyword cluster “vlms · visual reasoning · chain”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 5 of 6 tracked benchmarks.
24 papers in 2023–24 and 240 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 11.6% to 11.1%.
Open questions the data points to
- MM-Vet (score): best went from 45 (up to 2024) to 77.15 in 2025 — Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · 0 citations)
- MMBench (score): best went from 67.8 (up to 2024) to 80.8 in 2025 — Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · 1 citation)
- MME (score): best went from 1306 (up to 2024) to 2338 in 2025 — VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · 5 citations)
- GQA (accuracy): best went from 62.6 (up to 2024) to 76.14 in 2025 — Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update (AAAI 2025 · 2 citations)
- MathVista (score): best went from 37 (up to 2024) to 74.7 in 2025 — VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · 5 citations)
- Papers report LLaVA-1.5 on MME (score) anywhere from 1511 to 1808: comparisons against it depend on whose number you take.
What looks settled
- LLaVABench (score): best up to 2024 79.2, best in 2025–26 78.8 from 4 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · 75 citations)
- 2025 · FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · 58 citations)
- 2026 · Think Then Embed: Generative Context Improves Multimodal Embedding (ICLR 2026 · 2 citations)
Earliest papers here
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · 75 citations)
- VLP: Vision Language Planning for Autonomous Driving (CVPR 2024 · 74 citations)
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · 44 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MM-Vet · score (higher is better)
8 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 45 | Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · 5 citations) |
| 2025 | 77.15 | Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · 0 citations) |
MMBench · score (higher is better)
8 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 67.8 | Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · 5 citations) |
| 2025 | 80.8 | Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · 1 citation) |
MME · score (higher is better)
6 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 1306 | Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · 5 citations) |
| 2025 | 2338 | VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · 5 citations) |
GQA · accuracy (higher is better)
5 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 62.6 | Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · 5 citations) |
| 2025 | 76.14 | Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update (AAAI 2025 · 2 citations) |
LLaVABench · score (higher is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 79.2 | Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · 5 citations) |
MathVista · score (higher is better)
5 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 37 | Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · 5 citations) |
| 2025 | 74.7 | VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · 5 citations) |
What has been tried
109 papers here report 1,241 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 6%, hurt 24%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use SLP without ALP | nuScenes motion forecasting · minADE (m) | 0.71 -> 0.72 | Hurt | VLP: Vision Language Planning for Autonomous Driving (CVPR 2024 · 74 citations) |
| Put harmful instruction and incitement content in a text-only prompt rather than a typographic image | SafeBench-Tiny · ASR | 92.00% -> 60.00% | Hurt | FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · 58 citations) |
| use point-then-count instead of count-only | CountBenchQA test · accuracy | 87.9 -> 89.4 | Helped | Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models (CVPR 2025 · 51 citations) |
| Add two deconvolution layers for feature refinement | COCO-2017 val · AP | 57.7 -> 66.8 | Helped | RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · 44 citations) |
| add scene-context retrieval to the initial segment scores | UCF-Crime · AUC (%) | 76.10 -> 84.53 (+8.43) | Helped | VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models (CVPR 2025 · 41 citations) |
| replace Llama-1-7B with Mistral-7B as the language backbone | VQAv2 4-shot average · Avg. score | 62.5 -> 67.6 | Helped | What matters when building vision-language models? (NeurIPS 2024 · 41 citations) |
| add visual CoT (bbox-crop second pass) versus direct answering | Visual CoT benchmark, average · average score | 0.443 -> 0.550 | Helped | Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · 36 citations) |
| add expert-model feedback to VILA-M3-3B | ChestX-ray14 classification · F1 | 48.4 -> 51.3 | Helped | VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge (CVPR 2025 · 29 citations) |
| Set α to 0 instead of 1 in the combined training objective | COCO · R@5 | 58.5 to 33.9 | Hurt | Vision-Language Models Do Not Understand Negation (CVPR 2025 · 20 citations) |
| Apply natural-language feedback training (RLAIF) | LLaVA Eval Average · GPT-4 score | 70.05 -> 74.42 | Helped | DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback (CVPR 2024 · 19 citations) |
| Add the instruction to focus on the image | VQAv2 · macro accuracy (%) | 75.75 -> 76.74 | Helped | Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · 12 citations) |
| Replace the textual trigger target with an OCR textual, visual, or combined trigger target | 8 jailbreak scenarios (Avg.) · ASR | 0.007 -> 0.849; 0.007 -> 0.849; 0.007 -> 0.870 | Helped | Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · 11 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| LLaVA-1.5 | MME | score | 1511 – 1808 | 3 papers, e.g. NeurIPS 2025 and AAAI 2025 |
| LLaVA-1.5 | MMBench | score | 59.1 – 64.3 | 3 papers, e.g. AAAI 2025 and NeurIPS 2025 |
| LLaVA-1.5 7B | MME Cognition | score | 316.1 – 357.86 | 2 papers, e.g. NeurIPS 2025 and ICLR 2025 |
What reviewers will expect
Datasets and baselines named in the 240 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO · 19 of 240 papers
- VQAv2 · 14 of 240 papers
- MME · 13 of 240 papers
- MathVista · 13 of 240 papers
- MMBench · 13 of 240 papers
- MM-SafetyBench · 12 of 240 papers
- MMMU · 12 of 240 papers
- POPE · 11 of 240 papers
Baselines
- GPT-4o · 61 of 240 papers
- Gemini-2.5-Pro · 27 of 240 papers
- Qwen2.5-VL-7B · 19 of 240 papers
- Gemini 1.5 Pro · 17 of 240 papers
- LLaVA-1.5 · 15 of 240 papers
- Claude-3.5-Sonnet · 15 of 240 papers
- GPT-5 · 15 of 240 papers
- Qwen2.5-VL · 14 of 240 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 264 papers; from OpenAlex and the papers' affiliations).
- Shanghai AI Laboratory · 8 papers
- Stanford University · 7 papers
- The Chinese University of Hong Kong · 6 papers
- Peking University · 6 papers
- Tsinghua University · 5 papers
- Mohamed bin Zayed University of Artificial Intelligence · 5 papers
- University of Illinois, Urbana Champaign · 5 papers
- NVIDIA · 5 papers
- Microsoft · 5 papers
- Harbin Institute of Technology · 4 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Chain-of-thought multimodal reasoning (332 papers)
- Multimodal LLM benchmarks (252 papers)
- Spatial reasoning in VLMs (207 papers)
- Reinforcement learning for visual reasoning (163 papers)
Elsewhere in the topic
- Instruction-tuned multimodal assistants (199 papers)
- Benchmarks and analysis of MLLMs (181 papers)
- Interleaved image-text generation with MLLMs (153 papers)
- Unified any-modality language models (152 papers)
- Attention-based visual token pruning (117 papers)
- Decoding-time hallucination mitigation (91 papers)
- Training-free token reduction for MLLMs (90 papers)
- Visual instruction tuning (68 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).