Research map / Vision-language and multimodal / Multimodal LLMs
Instruction-tuned multimodal assistants
Builds stronger MLLMs through better visual encoders, small language models, and fine-grained visual knowledge or preference training.
199 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Multimodal LLMs, keyword cluster “mllm · instruction · llms”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 4 of 6 tracked benchmarks.
35 papers in 2023–24 and 164 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 16.9% to 7.6%.
Open questions the data points to
- MM-Vet (score): best went from 54.9 (up to 2024) to 79.6 in 2025 — NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · 0 citations)
- MME (score): best went from 1950 (up to 2024) to 2352 in 2025 — VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · 9 citations)
- GQA (accuracy): best went from 64.8 (up to 2024) to 66.3 in 2025 — Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · 4 citations)
- MMBench (score): best went from 70.4 (up to 2024) to 81.5 in 2025 — FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation (NeurIPS 2025 · 0 citations)
- Papers report VideoChat2 on MVBench (AVG) anywhere from 51.1 to 60.4: comparisons against it depend on whose number you take.
What looks settled
- POPE (score): best up to 2024 88.8, best in 2025–26 87.8 from 4 papers, at most 0.2% better, if at all.
- TextVQA (accuracy): best up to 2024 76.4, best in 2025–26 71.2 from 6 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · 166 citations)
- 2025 · LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding (AAAI 2025 · 36 citations)
- 2026 · MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · 2 citations)
Earliest papers here
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · 166 citations)
- Grounding Multimodal Large Language Models to the World (ICLR 2024 · 131 citations)
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024 · 111 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MM-Vet · score (higher is better)
12 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 54.9 | InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD (NeurIPS 2024 · 7 citations) |
| 2025 | 79.6 | NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · 0 citations) |
MME · score (higher is better)
11 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 1950 | Honeybee: Locality-enhanced Projector for Multimodal LLM (CVPR 2024 · 62 citations) |
| 2025 | 2352 | VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · 9 citations) |
GQA · accuracy (higher is better)
9 papers, 2024–2026 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 64.8 | MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · 10 citations) |
| 2025 | 66.3 | Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · 4 citations) |
MMBench · score (higher is better)
9 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 70.4 | MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · 10 citations) |
| 2025 | 81.5 | FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation (NeurIPS 2025 · 0 citations) |
POPE · score (higher is better)
8 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 88.8 | VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · 11 citations) |
TextVQA · accuracy (higher is better)
7 papers, 2024–2026 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 76.4 | MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · 10 citations) |
What has been tried
103 papers here report 1,193 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 5%, hurt 23%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Increase image resolution from 224² to 336² with I-MoF | MMVP · accuracy | 28.0 -> 31.3 | Helped | Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · 166 citations) |
| add multi-scale representation (vs single-scale) | GPT image understanding (COCO) · All score | 84.2 vs 74.2 | Helped | Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024 · 111 citations) |
| add [REJ] token for empty targets | gRefCOCO validation · N-acc. | 30.32 -> 56.45 | Helped | GSVA: Generalized Segmentation via Multimodal Large Language Models (CVPR 2024 · 71 citations) |
| Add V* visual search and visual working memory | V*Bench overall · accuracy | 45.02 -> 75.39 | Helped | V?: Guided Visual Search as a Core Mechanism in Multimodal LLMs (CVPR 2024 · 64 citations) |
| Increase visual tokens from M=144 to M=256 | SEED-Bench Image-only · accuracy | 64.5 -> 65.5 | Helped | Honeybee: Locality-enhanced Projector for Multimodal LLM (CVPR 2024 · 62 citations) |
| Add short-form response prompts and data to training | LVIS and PACO · SS and S-IoU | +8.83 SS and +12.54 S-IoU on LVIS; +22.80 SS on PACO | Helped | Osprey: Pixel Understanding with Visual Instruction Tuning (CVPR 2024 · 59 citations) |
| Use stage-wise rather than single-stage instruction tuning | REC Avg. · accuracy | 3.78 -> 54.46 | Helped | LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · 43 citations) |
| Add LoRA to instruction tuning | MAPLM-QA test · QNS | 82.13 -> 85.52 | Helped | MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · 43 citations) |
| Add grounding data to referring training | Referring object classification · performance | Helped | Ferret: Refer and Ground Anything Anywhere at Any Granularity (ICLR 2024 · 43 citations) | |
| Add multi-layer aggregation to SVA | OCR & Chart · category score | 55.2 -> 55.5 | Helped | Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · 37 citations) |
| add captioning pretraining | nuScenes-QA · accuracy | 41.2 -> 47.4 | Helped | LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding (AAAI 2025 · 36 citations) |
| Use Mask2Former+ trained on the combined multi-grained segmentation dataset instead of Mask2Former | RefCOCO+, PhraseCut, ReasonSeg, TextVQA-X · Any-IoU | 0.867 -> 0.873; 0.563 -> 0.624; 0.602 -> 0.745; 0.137 -> 0.446 | Mixed | GROUNDHOG: Grounding Large Language Models to Holistic Segmentation (CVPR 2024 · 30 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| VideoChat2 | MVBench | AVG | 51.1 – 60.4 | 2 papers, e.g. ICML 2024 and CVPR 2025 |
What reviewers will expect
Datasets and baselines named in the 164 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- DocVQA · 10 of 164 papers
- RefCOCO · 10 of 164 papers
- ChartQA · 9 of 164 papers
- GQA · 8 of 164 papers
- ShareGPT4V · 8 of 164 papers
- COCO · 8 of 164 papers
- POPE · 7 of 164 papers
- Visual Genome · 7 of 164 papers
Baselines
- GPT-4o · 38 of 164 papers
- LLaVA-1.5 · 20 of 164 papers
- Gemini-2.5-Flash · 14 of 164 papers
- Gemini-2.5-Pro · 13 of 164 papers
- InstructBLIP · 12 of 164 papers
- Gemini-1.5 Pro · 11 of 164 papers
- Qwen-VL-Chat · 10 of 164 papers
- LLaVA-NeXT · 10 of 164 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 199 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 13 papers
- The Chinese University of Hong Kong · 10 papers
- Peking University · 10 papers
- Shanghai Jiao Tong University · 8 papers
- Alibaba Group · 8 papers
- Chinese Academy of Sciences · 8 papers
- Nanjing University · 6 papers
- SenseTime · 6 papers
- Shanghai Artificial Intelligence Laboratory · 5 papers
- ByteDance · 5 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Benchmarks and analysis of MLLMs (181 papers)
- Unified any-modality language models (152 papers)
- Interleaved image-text generation with MLLMs (153 papers)
- Visual instruction tuning (68 papers)
Elsewhere in the topic
- Chain-of-thought multimodal reasoning (332 papers)
- VLM capability analysis (264 papers)
- Multimodal LLM benchmarks (252 papers)
- Spatial reasoning in VLMs (207 papers)
- Reinforcement learning for visual reasoning (163 papers)
- Attention-based visual token pruning (117 papers)
- Decoding-time hallucination mitigation (91 papers)
- Training-free token reduction for MLLMs (90 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).