Research map / Vision-language and multimodal
Human and embodied modeling: research map
2,095 papers on Human and embodied modeling in Vision-language and multimodal, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 5 clusters and 21 approaches. The busiest year so far is 2025.
What it is
Models that connect vision and language in human-centred ways: captioning, grounding phrases in images, retrieval, decoding what people see from brain scans, and comparing model perception with human perception.
Terms you will see on this page:
- Visual grounding
- locating the image region a phrase refers to.
- Image captioning
- describing an image in a sentence.
- Vision-language pretraining
- pretraining on image-text pairs to learn joint representations.
Start reading with the most cited papers here:
- Deep contextualized word representations (ICLR 2018 · 12,263 citations)
- BERTScore: Evaluating Text Generation with BERT (ICLR 2020 · 2,397 citations)
Within Vision-language and multimodal, its share shrank from 33.5% in 2023–24 to 14.0% in 2025–26 (532 → 669 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Human and embodied modeling in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- fMRI visual decoding rising: 4% → 10% of the topic (23 → 68 papers)
- Medical vision-language models rising: 8% → 12% of the topic (40 → 78 papers)
- Composed and text-based image retrieval rising: 3% → 6% of the topic (16 → 43 papers)
- Human-like visual cognition in VLMs falling: 8% → 4% of the topic (44 → 30 papers)
- Multimodal and affective interaction modeling falling: 7% → 4% of the topic (39 → 27 papers)
Papers from 2025–2026 that are already cited most:
- ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval (AAAI 2025 · 28 citations)
- Design2GarmentCode: Turning Design Concepts to Tangible Garments Through Program Synthesis (CVPR 2025 · 12 citations)
- Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior (CVPR 2025 · 10 citations)
- Hybrid Concept Bottleneck Models (CVPR 2025 · 6 citations)
Approaches and key papers
Retrieval, captioning and scene graphs · 776 papers
Keywords: retrieval, scene, captioning
Approaches in this cluster:
- Scene graph generation (153 papers)
Generates open-vocabulary and training-free scene graphs using vision-language models and object-centric binding. - Language-grounded scene text and structure (190 papers)
Connects images with text through scene text recognition, language bias correction, and scene-language representations. - Multimodal retrieval-augmented modeling (95 papers)
Uses cross-modal alignment, retrieval, and relation propagation for multimodal dialog and language modeling. - Composed and text-based image retrieval (122 papers)
Retrieves images from text, sketch, or reference-plus-modification queries with vision-language encoders. - Concept-guided image captioning (94 papers)
Improves captions by injecting semantic concepts, context guidance, and scene-graph evaluation. - Neural image captioning (122 papers)
Generate descriptive, controllable, and personalized captions from images.
Most cited and most cited since 2024:
- Image Captioning With Semantic Attention (CVPR 2016 · 1,739 citations)
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning (CVPR 2017 · 1,734 citations)
- HIPTrack: Visual Tracking with Historical Prompts (CVPR 2024 · 105 citations)
- UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity (CVPR 2024 · 68 citations)
Human perception and cognition · 463 papers
Keywords: computer vision, humans, abstract
Approaches in this cluster:
- Multimodal and affective interaction modeling (95 papers)
Models multimodal interactions, emotions, and social cues with datasets and information decomposition. - Human-like visual cognition in VLMs (126 papers)
Tests and improves vision-language models on human visual cognition, compositionality, and structure extraction. - Human perception modeling (109 papers)
Models human visual motion, saliency, and eye movement with trainable networks and human signals. - Abstract visual reasoning benchmarks (70 papers)
Benchmarks few-shot and systematic visual reasoning using object-centric abstraction and analogical learning. - Scientific domain vision datasets (63 papers)
Builds large-scale datasets for ecological, geological, and agricultural imagery plus physical prediction.
Most cited and most cited since 2024:
- VMamba: Visual State Space Model (NeurIPS 2024 · 830 citations)
- Detecting Visual Relationships With Deep Relational Networks (CVPR 2017 · 507 citations)
- BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · 124 citations)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · 78 citations)
Vision-language pretraining · 462 papers
Keywords: pre, generation, unified
Approaches in this cluster:
- Vision-language pretraining (203 papers)
Pretrains unified vision-language models using image-as-language tokens and decoupled visual encoding. - Medical vision-language models (139 papers)
Pretrains medical vision-language models for radiology reports and 3D imaging. - Layout-aware document understanding (82 papers)
Extracts information from visually rich documents with layout-aware pretrained models. - Text-driven human motion generation (38 papers)
Generates human motion and interactions from text using language-aligned motion models.
Most cited and most cited since 2024:
- Deep contextualized word representations (ICLR 2018 · 12,263 citations)
- BEiT: BERT Pre-Training of Image Transformers (ICLR 2022 · 915 citations)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · 251 citations)
- Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction (CVPR 2024 · 143 citations)
Brain decoding and visual concepts · 235 papers
Keywords: concepts, brain, fmri
Approaches in this cluster:
- fMRI visual decoding (117 papers)
Decodes and synthesizes visual stimuli from brain recordings using semantic and cross-subject models. - Language-grounded visual concept learning (97 papers)
Learns visual concepts by tokenization, decomposition, and transfer from language models. - Concept bottleneck models (21 papers)
Builds interpretable classifiers that predict human-understandable concepts using vision-language guidance.
Most cited and most cited since 2024:
- Dense Captioning With Joint Inference and Visual Context (CVPR 2017 · 175 citations)
- PaStaNet: Toward Human Activity Knowledge Engine (CVPR 2020 · 145 citations)
- MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment (AAAI 2024 · 27 citations)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · 21 citations)
Visual grounding · 159 papers
Keywords: grounding, referring, expression
Approaches in this cluster:
- Visual grounding transformers (81 papers)
Localizes language queries in images and 3D scenes with decoupled fusion and verification. - Region-level vision-language alignment (40 papers)
Aligns regions and objects with language through visual experts and hierarchical grounded annotation. - Referring expression comprehension (38 papers)
Resolves referring expressions with one-step transformers, iterative scanning, and adaptive inference.
Most cited and most cited since 2024:
- MAttNet: Modular Attention Network for Referring Expression Comprehension (CVPR 2018 · 869 citations)
- Modeling Relationships in Referential Expressions With Compositional Modular Networks (CVPR 2017 · 410 citations)
- GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · 175 citations)
- CogVLM: Visual Expert for Pretrained Language Models (NeurIPS 2024 · 77 citations)
Related topics in Vision-language and multimodal
- Visual question answering (711)
- Multimodal LLMs (3,132)
- Robot action and navigation (841)
- CLIP and zero-shot (1,294)
- Cross-modal learning (1,766)
