Research map / Vision-language and multimodal / Cross-modal learning
Unimodal-to-multimodal representation transfer
Studies cross-modal misalignment, distillation and dataset condensation to learn across modalities from unimodal data.
145 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Cross-modal learning, keyword cluster “cross · retrieval · image text”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
35 papers in 2023–24 and 93 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 12.1% to 11.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Cross Modal Distillation for Supervision Transfer (CVPR 2016 · 555 citations)
- 2017 · Learning Cross-Modal Deep Representations for Robust Pedestrian Detection (CVPR 2017 · 241 citations)
- 2018 · Spatiotemporal Activity Modeling Under Data Scarcity: A Graph-Regularized Cross-Modal Embedding Approach (AAAI 2018 · 15 citations)
- 2019 · ContextDesc: Local Descriptor Augmentation With Cross-Modality Context (CVPR 2019 · 233 citations)
- 2020 · Semi-Supervised Multi-Modal Learning with Balanced Spectral Decomposition (AAAI 2020 · 24 citations)
- 2021 · Multimodal Contrastive Training for Visual Representation Learning (CVPR 2021 · 172 citations)
- 2022 · AXM-Net: Implicit Cross-Modal Feature Alignment for Person Re-identification (AAAI 2022 · 140 citations)
- 2023 · Bridging Search Region Interaction With Template for RGB-T Tracking (CVPR 2023 · 147 citations)
- 2024 · Noisy-Correspondence Learning for Text-to-Image Person Re-identification (CVPR 2024 · 126 citations)
- 2025 · MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation (AAAI 2025 · 47 citations)
- 2026 · Distilling Cross-Modal Knowledge via Feature Disentanglement (AAAI 2026 · 5 citations)
Earliest papers here
- Cross Modal Distillation for Supervision Transfer (CVPR 2016 · 555 citations)
- Learning Aligned Cross-Modal Representations From Weakly Aligned Data (CVPR 2016 · 197 citations)
- Learning Cross-Modal Deep Representations for Robust Pedestrian Detection (CVPR 2017 · 241 citations)
What has been tried
79 papers here report 905 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 4%, hurt 19%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace TAL with TRL | CUHK-PEDES (50% noise) · Rank-1 accuracy | 71.33 -> 6.40 | Hurt | Noisy-Correspondence Learning for Text-to-Image Person Re-identification (CVPR 2024 · 126 citations) |
| add cross-image correlation awareness to AdaptSPM | Pitts30k · R@1 | 90.6 -> 94.8 | Helped | CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition (CVPR 2024 · 78 citations) |
| add general feature enhancement | Baby, Sports, Clothing · Recall@20 | 0.1011 -> 0.1048; 0.1094 -> 0.1139; 0.0949 -> 0.0989 | Helped | MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation (AAAI 2025 · 47 citations) |
| Replace KL loss with DIST loss | NYU-Depth V2 RGB, ResNet-18/ResNet-18 · mIoU (%) | 37.5 -> 38.1 | Helped | C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation (CVPR 2024 · 44 citations) |
| Add CrossMMD during training | NTU60 cross-subject, CTR-GCN · O-AUROC | 86.23 -> 88.31 | Helped | Navigating Open Set Scenarios for Skeleton-Based Action Recognition (AAAI 2024 · 29 citations) |
| Add central alignment strategy | MIMIC-CXR test set · ROUGE-L | 0.261±0.013 -> 0.274±0.012 | Helped | MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant (CVPR 2024 · 28 citations) |
| Add the conventional classification head | Market1501, R→R · Rank-1 | 3.2 -> 78.0 | Helped | All in One Framework for Multimodal Re-identification in the Wild (CVPR 2024 · 26 citations) |
| Add OT-based local alignment | CMU-MOSI · Accuracy | 84.6 -> 86.9 | Helped | AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · 18 citations) |
| Add matching regularization LESC to ESC | Flickr30K, 40% noise · R@1 image-to-text | 73.0 -> 76.1 | Helped | Robust Noisy Correspondence Learning with Equivariant Similarity Consistency (CVPR 2024 · 17 citations) |
| Fix curvature at 0.5 or higher rather than allowing MERU’s learned curvature | ImageNet zero-shot · accuracy | Hurt | Accept the Modality Gap: An Exploration in the Hyperbolic Space (CVPR 2024 · 16 citations) | |
| add attribute-based texts to class name-based texts | Caltech101; Oxford Pets; Describable Textures; EuroSAT · accuracy | 79.90 -> 89.80; 81.63 -> 85.91; 41.28 -> 45.27; 44.89 -> 46.57 | Helped | Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification (CVPR 2024 · 14 citations) |
| Combine self-agent and cross-agent relationships | Boreas; MinkLoc++3D teacher → MinkLoc++2D · AR@1 | 61.3 -> 68.2 | Helped | DistilVPR: Cross-Modal Knowledge Distillation for Visual Place Recognition (AAAI 2024 · 12 citations) |
What reviewers will expect
Datasets and baselines named in the 41 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Flickr30K · 5 of 41 papers
- ImageNet · 4 of 41 papers
- MS-COCO · 4 of 41 papers
- AudioCaps · 3 of 41 papers
- RGBT234 · 2 of 41 papers
- Amazon Baby · 2 of 41 papers
- Amazon Sports and Outdoors · 2 of 41 papers
- Amazon Clothing, Shoes and Jewelry · 2 of 41 papers
Baselines
- CLIP · 6 of 41 papers
- CSA · 2 of 41 papers
- random noise patch · 2 of 41 papers
- LightGCN · 2 of 41 papers
- LayerGCN · 2 of 41 papers
- VBPR · 2 of 41 papers
- DualGNN · 2 of 41 papers
- LATTICE · 2 of 41 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 128 papers; from OpenAlex and the papers' affiliations).
- Carnegie Mellon University · 5 papers
- Shanghai Jiao Tong University · 5 papers
- University of Chinese Academy of Sciences · 4 papers
- Central South University · 4 papers
- Hong Kong University of Science and Technology · 4 papers
- Chinese Academy of Sciences · 3 papers
- The Hong Kong Polytechnic University · 3 papers
- The Chinese University of Hong Kong · 3 papers
- Peng Cheng Laboratory · 3 papers
- Xidian University · 3 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Probabilistic embeddings for cross-modal retrieval (108 papers)
- Contrastive vision-language alignment (104 papers)
- Cross-modal hashing (45 papers)
Elsewhere in the topic
- Multimodal fusion with missing modalities (208 papers)
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Multimodal sentiment analysis (106 papers)
- Audio-visual speech separation (76 papers)
- Audio-visual source localization (67 papers)
- Self-supervised audio-visual representation learning (44 papers)
- Multi-modal object re-identification (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).