Research map / Vision-language and multimodal
Multimodal LLMs: research map
3,132 papers on Multimodal LLMs in Vision-language and multimodal, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 4 clusters and 15 approaches. The busiest year so far is 2026.
What it is
Multimodal large language models (MLLMs) connect an image or video encoder to a language model, so one model can describe, reason about and chat about visual input. Research covers instruction tuning, benchmarks, reducing visual tokens for speed, visual reasoning and hallucination.
Terms you will see on this page:
- MLLM / LVLM
- a multimodal or large vision-language model that takes images and text.
- Visual tokens
- the image features fed to the language model as a token sequence.
- Hallucination
- describing objects or details that are not in the image.
Start reading with the most cited papers here:
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · 1,490 citations)
- Visual Instruction Tuning (NeurIPS 2023 · 1,198 citations)
Within Vision-language and multimodal, its share grew from 13.0% in 2023–24 to 45.1% in 2025–26 (207 → 2,156 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Multimodal LLMs in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Reinforcement learning for visual reasoning rising: 0% → 8% of the topic (1 → 162 papers)
- Chain-of-thought multimodal reasoning rising: 8% → 14% of the topic (17 → 312 papers)
- Spatial reasoning in VLMs rising: 3% → 9% of the topic (7 → 198 papers)
- Visual instruction tuning falling: 12% → 2% of the topic (24 → 44 papers)
- Instruction-tuned multimodal assistants falling: 17% → 8% of the topic (35 → 164 papers)
Papers from 2025–2026 that are already cited most:
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · 119 citations)
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (CVPR 2025 · 92 citations)
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · 58 citations)
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models (CVPR 2025 · 51 citations)
Approaches and key papers
Visual reasoning in VLMs · 1,633 papers
Keywords: vlms, visual reasoning, chain
Approaches in this cluster:
- VLM capability analysis (378 papers)
Probes and refines vision-language models, assessing priors, reasoning, and design choices. - Chain-of-thought multimodal reasoning (425 papers)
Improves multimodal LLM reasoning with latent thinking, long chains, and structured scene-graph reasoning. - Multimodal LLM benchmarks (332 papers)
Builds comprehensive and evolving benchmarks for large multimodal models. - Spatial reasoning in VLMs (307 papers)
Improves grounded spatial reasoning and perspective understanding in vision-language models. - Reinforcement learning for visual reasoning (191 papers)
Trains vision-language models with reinforcement learning and data augmentation to reason visually.
Most cited and most cited since 2024:
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (CVPR 2024 · 355 citations)
- SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (CVPR 2024 · 234 citations)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · 98 citations)
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (CVPR 2025 · 92 citations)
Multimodal assistants and instruction tuning · 965 papers
Keywords: mllm, instruction, llms
Approaches in this cluster:
- Instruction-tuned multimodal assistants (269 papers)
Builds stronger MLLMs through better visual encoders, small language models, and fine-grained visual knowledge or preference training. - Benchmarks and analysis of MLLMs (210 papers)
Evaluates and probes multimodal LLMs, covering feature fusion, information flow, benchmarks, and jailbreak weaknesses. - Unified any-modality language models (183 papers)
Aligns audio, video and other modalities with LLMs in one framework while avoiding text-only forgetting. - Interleaved image-text generation with MLLMs (222 papers)
Uses multimodal LLMs to jointly understand and generate images and text in context. - Visual instruction tuning (81 papers)
Fine-tunes language models on image-instruction data, generated or filtered, to follow multimodal instructions.
Most cited and most cited since 2024:
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · 1,490 citations)
- Visual Instruction Tuning (NeurIPS 2023 · 1,198 citations)
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (ICLR 2024 · 480 citations)
- CogAgent: A Visual Language Model for GUI Agents (CVPR 2024 · 173 citations)
Visual token reduction · 297 papers
Keywords: tokens, pruning, visual token
Approaches in this cluster:
- Attention-based visual token pruning (173 papers)
Drops redundant visual tokens inside VLMs using attention or text-guided importance scores to speed up inference. - Training-free token reduction for MLLMs (124 papers)
Searches or selects which vision tokens to trim or merge across layers and frames without retraining.
Most cited and most cited since 2024:
- VisionZip: Longer is Better but Not Necessary in Vision Language Models (CVPR 2025 · 46 citations)
- NVILA: Efficient Frontier Visual Language Models (CVPR 2025 · 37 citations)
- FastVLM: Efficient Vision Encoding for Vision Language Models (CVPR 2025 · 27 citations)
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · 27 citations)
Hallucination in vision-language models · 237 papers
Keywords: hallucination, lvlms, mitigating
Approaches in this cluster:
- Decoding-time hallucination mitigation (108 papers)
Alters decoding, such as contrastive or layer-wise decoding, to reduce object hallucination in vision-language models. - Preference tuning against hallucination (70 papers)
Trains models with preference optimization and targeted data to suppress hallucinated content and locate its causes. - Bias-aware multi-image robustness (59 papers)
Analyzes language and position biases in LVLMs and corrects them through selective training or attention perturbation.
Most cited and most cited since 2024:
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · 172 citations)
- Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · 145 citations)
- HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (CVPR 2024 · 138 citations)
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (CVPR 2024 · 124 citations)
Related topics in Vision-language and multimodal
- Human and embodied modeling (2,095)
- Visual question answering (711)
- Robot action and navigation (841)
- CLIP and zero-shot (1,294)
- Cross-modal learning (1,766)
