Research map / Vision-language and multimodal
Cross-modal learning: research map
1,766 papers on Cross-modal learning in Vision-language and multimodal, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 3 clusters and 12 approaches. The busiest year so far is 2026.
What it is
Learning from several modalities at once (images, text, audio, depth, medical scans): fusing them, coping when one is missing, retrieving across them, and using sound to locate or separate sources in video.
Terms you will see on this page:
- Multimodal fusion
- combining information from several input types.
- Cross-modal retrieval
- finding items in one modality (images) from a query in another (text).
- Audio-visual learning
- using sound and video together, such as locating what makes a sound.
Start reading with the most cited papers here:
- Deep Cross-Modal Hashing (CVPR 2017 · 898 citations)
- ImageBind: One Embedding Space To Bind Them All (CVPR 2023 · 810 citations)
Within Vision-language and multimodal, its share held steady from 18.3% in 2023–24 to 16.5% in 2025–26 (290 → 788 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Cross-modal learning in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Medical multimodal fusion rising: 10% → 15% of the topic (29 → 121 papers)
- Multimodal sentiment analysis rising: 6% → 9% of the topic (18 → 73 papers)
- Cross-modal hashing rising: 2% → 3% of the topic (5 → 25 papers)
- Self-supervised audio-visual representation learning falling: 6% → 2% of the topic (18 → 12 papers)
- Multimodal fusion with missing modalities falling: 18% → 16% of the topic (52 → 126 papers)
Papers from 2025–2026 that are already cited most:
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · 77 citations)
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content (AAAI 2025 · 49 citations)
- BSAFusion: A Bidirectional Stepwise Feature Alignment Network for Unaligned Medical Image Fusion (AAAI 2025 · 44 citations)
- Vision-guided Text Mining for Unsupervised Cross-modal Hashing with Community Similarity Quantization (AAAI 2025 · 10 citations)
Approaches and key papers
Multimodal fusion · 945 papers
Keywords: fusion, missing, multi modal
Approaches in this cluster:
- Multimodal fusion with missing modalities (258 papers)
Learns aligned multimodal representations robust to missing or imbalanced modalities. - Multi-modality image fusion (305 papers)
Fuses RGB, infrared, event and other modalities with transformers and semantic priors. - Medical multimodal fusion (203 papers)
Combines clinical and imaging modalities with decomposition, symmetric consistency and graph structure. - Multimodal sentiment analysis (121 papers)
Uses distillation and consistency learning for sentiment recognition, including with missing modalities. - Multi-modal object re-identification (58 papers)
Selects tokens and decouples features across RGB, infrared and text for re-identification.
Most cited and most cited since 2024:
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (AAAI 2021 · 751 citations)
- What Makes Training Multi-Modal Classification Networks Hard? (CVPR 2020 · 532 citations)
- Equivariant Multi-Modality Image Fusion (CVPR 2024 · 233 citations)
- Bi-directional Adapter for Multimodal Tracking (AAAI 2024 · 132 citations)
Cross-modal retrieval · 533 papers
Keywords: cross, retrieval, image text
Approaches in this cluster:
- Unimodal-to-multimodal representation transfer (175 papers)
Studies cross-modal misalignment, distillation and dataset condensation to learn across modalities from unimodal data. - Probabilistic embeddings for cross-modal retrieval (156 papers)
Matches images and text using probabilistic, multi-embedding or generative-aided representations. - Contrastive vision-language alignment (139 papers)
Refines contrastive and preference-based alignment between image and text embeddings. - Cross-modal hashing (63 papers)
Learns compact binary codes for efficient cross-modal retrieval.
Most cited and most cited since 2024:
- Deep Cross-Modal Hashing (CVPR 2017 · 898 citations)
- ImageBind: One Embedding Space To Bind Them All (CVPR 2023 · 810 citations)
- Noisy-Correspondence Learning for Text-to-Image Person Re-identification (CVPR 2024 · 126 citations)
- CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition (CVPR 2024 · 78 citations)
Audio-visual learning · 288 papers
Keywords: audio, sound, speech
Approaches in this cluster:
- Audio-visual source localization (122 papers)
Aligns sound and visual scenes to localize and segment sounding objects. - Audio-visual speech separation (97 papers)
Uses visual cues such as lips and cross-modal prompts to separate speech and adapt pretrained models. - Self-supervised audio-visual representation learning (69 papers)
Learns audio-visual features from correspondence and synchronization via contrastive and distillation objectives.
Most cited and most cited since 2024:
- Balanced Multimodal Learning via On-the-Fly Gradient Modulation (CVPR 2022 · 336 citations)
- Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching (CVPR 2018 · 240 citations)
- AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection (CVPR 2024 · 76 citations)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · 67 citations)
Related topics in Vision-language and multimodal
- Human and embodied modeling (2,095)
- Visual question answering (711)
- Multimodal LLMs (3,132)
- Robot action and navigation (841)
- CLIP and zero-shot (1,294)
