Research map / Vision-language and multimodal / Cross-modal learning
Probabilistic embeddings for cross-modal retrieval
Matches images and text using probabilistic, multi-embedding or generative-aided representations.
108 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Cross-modal learning, keyword cluster “cross · retrieval · image text”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Few recent reports The tracked benchmarks have at most one report from 2025–26 that could beat the old record, too few to say whether they are still moving.
21 papers in 2023–24 and 48 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 7.2% to 6.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Aggregating Image and Text Quantized Correlated Components (CVPR 2016 · 25 citations)
- 2017 · Learning Cross-Modal Embeddings for Cooking Recipes and Food Images (CVPR 2017 · 421 citations)
- 2018 · Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval With Generative Models (CVPR 2018 · 421 citations)
- 2019 · Cross-Modal Self-Attention Network for Referring Image Segmentation (CVPR 2019 · 534 citations)
- 2020 · Multi-Modality Cross Attention Network for Image and Sentence Matching (CVPR 2020 · 453 citations)
- 2021 · Probabilistic Embeddings for Cross-Modal Retrieval (CVPR 2021 · 253 citations)
- 2022 · Cross Modal Retrieval With Querybank Normalisation (CVPR 2022 · 99 citations)
- 2023 · Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval (CVPR 2023 · 320 citations)
- 2024 · Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (AAAI 2024 · 52 citations)
- 2025 · Chat-based Person Retrieval via Dialogue-Refined Cross-Modal Alignment (CVPR 2025 · 13 citations)
- 2026 · Hierarchical Prompt Learning for Image- and Text-Based Person Re-Identification (AAAI 2026 · 10 citations)
Earliest papers here
- Aggregating Image and Text Quantized Correlated Components (CVPR 2016 · 25 citations)
- Learning Cross-Modal Embeddings for Cooking Recipes and Food Images (CVPR 2017 · 421 citations)
- Online Asymmetric Similarity Learning for Cross-Modal Retrieval (CVPR 2017 · 44 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MS-COCO 5K · rSum (higher is better)
4 papers, 2020–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 416.5 | IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval (CVPR 2020 · 434 citations) |
| 2023 | 450.9 | Learning Semantic Relationship Among Instances for Image-Text Matching (CVPR 2023 · 115 citations) |
What has been tried
72 papers here report 793 measured changes (a component added, removed or swapped, compared with a baseline): helped 81%, no clear effect 3%, hurt 14%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add UMF | RSTPReid · R@1 | 68.15 -> 69.55 | Helped | Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (AAAI 2024 · 52 citations) |
| Add CSA alignment | MSCOCO · RSUM | 425.7 -> 429.7 | Helped | Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval (AAAI 2024 · 44 citations) |
| Add MUM | RSTPReid · Rank-1 accuracy | 59.80 -> 62.35 | Helped | Unifying Multi-Modal Uncertainty Modeling and Semantic Alignment for Text-to-Image Person Re-identification (AAAI 2024 · 39 citations) |
| Add partial OT | Flickr30K, MRate 0.6 · Image-to-Text R@1 | 62.2 -> 65.1 | Helped | Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024 · 38 citations) |
| Add the Reverse-Query objective | FashionIQ validation · R@1 | 19.98 -> 20.69 | Helped | Data Roaming and Quality Assessment for Composed Image Retrieval (AAAI 2024 · 36 citations) |
| Add the style intervener | CUHK-PEDES · Rank-1 | 64.04 -> 65.23 | Helped | Causality-Inspired Invariant Representation Learning for Text-Based Person Retrieval (AAAI 2024 · 31 citations) |
| add feature optimization to the generated pseudo-image feature baseline | MSCOCO in-domain · CIDEr | 88.5 -> 89.8 | Helped | Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · 21 citations) |
| apply data cleaning to generated dialogues | ChatPedes test · R-1 | 73.72 -> 75.67 | Helped | Chat-based Person Retrieval via Dialogue-Refined Cross-Modal Alignment (CVPR 2025 · 13 citations) |
| add multi-level alignment (instance-level plus word-level) | Multi30K · SumR | 473.2 -> 490.0; 482.3 -> 493.0; 472.6 -> 480.4 | Helped | CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer (AAAI 2024 · 13 citations) |
| Add consistency learning loss Lcl | NUS-WIDE-10K · mAP@all (Avg.) | 0.586 -> 0.598 | Helped | Fuzzy Multimodal Learning for Trusted Cross-modal Retrieval (CVPR 2025 · 9 citations) |
| Add R-GCN spatial visual encoding | TWITTER-2015 and TWITTER-2017 test · F1 | 75.07 -> 76.49; 86.05 -> 87.13 | Helped | Hierarchical Aligned Multimodal Learning for NER on Tweet Posts (AAAI 2024 · 9 citations) |
| add concept decomposition and its associated loss functions | QVHighlights val split · R1@0.5 | 59.35 -> 62.19 | Helped | CDTR: Semantic Alignment for Video Moment Retrieval Using Concept Decomposition Transformer (AAAI 2025 · 7 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MS-COCO · 6 of 17 papers
- Flickr30K · 6 of 17 papers
- Wikipedia · 2 of 17 papers
- INRIA-Websearch · 2 of 17 papers
- XMediaNet · 2 of 17 papers
- MSRVTT · 2 of 17 papers
- CUHK-PEDES · 2 of 17 papers
- M-BEIR · 2 of 17 papers
Baselines
- SCAN · 3 of 17 papers
- VSE∞ · 2 of 17 papers
- SGRAF · 2 of 17 papers
- NAAF · 2 of 17 papers
- MCCA · 2 of 17 papers
- SCL · 2 of 17 papers
- CLIP-SF · 2 of 17 papers
- BLIP-FF · 2 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 69 papers; from OpenAlex and the papers' affiliations).
- University of Science and Technology of China · 4 papers
- University of Electronic Science and Technology of China · 3 papers
- Xi’an Jiaotong University · 2 papers
- Zhejiang Gongshang University · 2 papers
- Southeast University · 2 papers
- Sichuan University · 2 papers
- Wuhan University · 2 papers
- Chinese academy of science · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Unimodal-to-multimodal representation transfer (145 papers)
- Contrastive vision-language alignment (104 papers)
- Cross-modal hashing (45 papers)
Elsewhere in the topic
- Multimodal fusion with missing modalities (208 papers)
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Multimodal sentiment analysis (106 papers)
- Audio-visual speech separation (76 papers)
- Audio-visual source localization (67 papers)
- Self-supervised audio-visual representation learning (44 papers)
- Multi-modal object re-identification (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).