Research map / Vision-language and multimodal / Multimodal LLMs
Unified any-modality language models
Aligns audio, video and other modalities with LLMs in one framework while avoiding text-only forgetting.
152 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Multimodal LLMs, keyword cluster “mllm · instruction · llms”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 3 of 5 tracked benchmarks.
21 papers in 2023–24 and 131 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.1% to 6.1%.
Open questions the data points to
- MM-Vet (score): best went from 29.1 (up to 2024) to 63.92 in 2025 — Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · 1 citation)
- VQAv2 (accuracy): best went from 71.6 (up to 2024) to 81.6 in 2025 — Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models (NeurIPS 2025 · 0 citations)
- VizWiz (accuracy): best went from 45.9 (up to 2024) to 56.2 in 2025 — Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference (AAAI 2025 · 46 citations)
What looks settled
- TextVQA (accuracy): best up to 2024 67, best in 2025–26 65.4 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · Language Is Not All You Need: Aligning Perception with Language Models (NeurIPS 2023 · 160 citations)
- 2024 · CogAgent: A Visual Language Model for GUI Agents (CVPR 2024 · 173 citations)
- 2025 · Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference (AAAI 2025 · 46 citations)
- 2026 · MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · 2 citations)
Earliest papers here
- Language Is Not All You Need: Aligning Perception with Language Models (NeurIPS 2023 · 160 citations)
- CogAgent: A Visual Language Model for GUI Agents (CVPR 2024 · 173 citations)
- OneLLM: One Framework to Align All Modalities with Language (CVPR 2024 · 81 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MM-Vet · score (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 29.1 | OneLLM: One Framework to Align All Modalities with Language (CVPR 2024 · 81 citations) |
| 2025 | 63.92 | Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · 1 citation) |
MME · score (higher is better)
4 papers, 2024–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 1640 | CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts (NeurIPS 2024 · 3 citations) |
TextVQA · accuracy (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 67 | CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts (NeurIPS 2024 · 3 citations) |
VQAv2 · accuracy (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 71.6 | OneLLM: One Framework to Align All Modalities with Language (CVPR 2024 · 81 citations) |
| 2025 | 81.6 | Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models (NeurIPS 2025 · 0 citations) |
VizWiz · accuracy (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 45.9 | OneLLM: One Framework to Align All Modalities with Language (CVPR 2024 · 81 citations) |
| 2025 | 56.2 | Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference (AAAI 2025 · 46 citations) |
What has been tried
64 papers here report 724 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 6%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add the high-resolution cross-module at 1120 resolution instead of using the original CogVLM structure at 490 resolution | ST-VQA · ANLS | 68.1 -> 78.2 | Helped | CogAgent: A Visual Language Model for GUI Agents (CVPR 2024 · 173 citations) |
| jointly train modalities rather than train each modality separately | ClothoQA · score | 37.8 -> 57.4 | Helped | OneLLM: One Framework to Align All Modalities with Language (CVPR 2024 · 81 citations) |
| use fused DINOv2 and SigLIP features instead of SigLIP-only features | VSR · accuracy | 53.2 -> 58.4 | Helped | Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference (AAAI 2025 · 46 citations) |
| Recurrent user preference inference (vs. direct inference over full history) | ablation (three datasets) · AUC/HR@5/MRR@5 | Helped | Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation (AAAI 2025 · 42 citations) | |
| Add FCA to TVP without FCA or TSE | CIFAR-10 · Avg. Δ | 0.28 -> 3.82 | Helped | Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · 17 citations) |
| switch from PaLI-X FT to PaLI-X LoRA as the SMoLA starting checkpoint | full-mixture benchmark average · average score gain | +1.26 vs +1.64 | Mixed | Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts (CVPR 2024 · 15 citations) |
| Add interleaved generative pretraining (I-GPT) stage with MMC4 data (ID setting) | MS-COCO text-to-image and comprehension synergy analysis · FID / comprehension | Helped | DreamLLM: Synergistic Multimodal Comprehension and Creation (ICLR 2024 · 15 citations) | |
| Gated attention with dynamic gate vs fixed gate values (g in [0,0.1]) | Averaged over MNER, MRE, MEE · F1 | dynamic gate beats any fixed gate; vs g=0 gains of 0.1 (MNER), 2.3 (MRE), 3.3 (MEE) | Helped | UMIE: Unified Multimodal Information Extraction with Instruction Tuning (AAAI 2024 · 13 citations) |
| add CoT pre-training | DocVQA · score | 76.00 -> 77.43 | Helped | DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · 12 citations) |
| Combine highlight guidance with attention activation | MME · score | 1528.7 -> 1552.5 | Helped | Prompt Highlighter: Interactive Control for Multi-Modal LLMs (CVPR 2024 · 11 citations) |
| Use k=10 sampled tokens and α=1.0 | Hyperparameter evaluation · ASR and MM-Vet utility | Helped | Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment (CVPR 2025 · 8 citations) | |
| Add token-level masking to DMK loss | Donald Trump · EM | 92.0±0.0 -> 99.0±0.0 | Helped | Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models (NeurIPS 2024 · 7 citations) |
What reviewers will expect
Datasets and baselines named in the 131 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- TextVQA · 6 of 131 papers
- MS-COCO · 6 of 131 papers
- AudioCaps · 6 of 131 papers
- COCO · 5 of 131 papers
- Flickr30k · 4 of 131 papers
- VGGSound · 4 of 131 papers
- ScienceQA · 3 of 131 papers
- VSR · 3 of 131 papers
Baselines
- GPT-4o · 30 of 131 papers
- Gemini-2.5-Flash · 13 of 131 papers
- Gemini-2.5-Pro · 11 of 131 papers
- Qwen2.5-VL-72B · 9 of 131 papers
- GPT-5 · 8 of 131 papers
- GPT-4V · 7 of 131 papers
- Qwen2.5-VL-7B · 7 of 131 papers
- InternVL3-8B · 7 of 131 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 152 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 9 papers
- University of Science and Technology of China · 8 papers
- Alibaba Group · 8 papers
- ByteDance · 6 papers
- Shanghai Jiao Tong University · 5 papers
- Chinese Academy of Sciences · 5 papers
- National University of Singapore · 5 papers
- Fudan University · 4 papers
- Hong Kong University of Science and Technology · 4 papers
- Tsinghua University · 4 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Instruction-tuned multimodal assistants (199 papers)
- Benchmarks and analysis of MLLMs (181 papers)
- Interleaved image-text generation with MLLMs (153 papers)
- Visual instruction tuning (68 papers)
Elsewhere in the topic
- Chain-of-thought multimodal reasoning (332 papers)
- VLM capability analysis (264 papers)
- Multimodal LLM benchmarks (252 papers)
- Spatial reasoning in VLMs (207 papers)
- Reinforcement learning for visual reasoning (163 papers)
- Attention-based visual token pruning (117 papers)
- Decoding-time hallucination mitigation (91 papers)
- Training-free token reduction for MLLMs (90 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).