Research map / Vision-language and multimodal / Cross-modal learning
Contrastive vision-language alignment
Refines contrastive and preference-based alignment between image and text embeddings.
104 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Cross-modal learning, keyword cluster “cross · retrieval · image text”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Few recent reports None of the tracked benchmarks has a report from 2025–26.
24 papers in 2023–24 and 67 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 8.3% to 8.5%.
No longer reported by papers here: SNLI-VE test (accuracy, last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Cross-Lingual Propagation for Deep Sentiment Analysis (AAAI 2018 · 38 citations)
- 2020 · Bi-Directional Relationship Inferring Network for Referring Image Segmentation (CVPR 2020 · 209 citations)
- 2021 · Learning the Best Pooling Strategy for Visual Semantic Embedding (CVPR 2021 · 284 citations)
- 2022 · FLAVA: A Foundational Language and Vision Alignment Model (CVPR 2022 · 532 citations)
- 2023 · ImageBind: One Embedding Space To Bind Them All (CVPR 2023 · 810 citations)
- 2024 · Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · 21 citations)
- 2025 · Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Large Model Enhancement (CVPR 2025 · 11 citations)
- 2026 · Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge (ICML 2026 · 0 citations)
Earliest papers here
- Cross-Lingual Propagation for Deep Sentiment Analysis (AAAI 2018 · 38 citations)
- Learning Multi-Modal Word Representation Grounded in Visual Context (AAAI 2018 · 30 citations)
- Bi-Directional Relationship Inferring Network for Referring Image Segmentation (CVPR 2020 · 209 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
SNLI-VE test · accuracy (higher is better)
4 papers, 2022–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 91.2 | OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (ICML 2022 · 251 citations) |
What has been tried
51 papers here report 615 measured changes (a component added, removed or swapped, compared with a baseline): helped 73%, no clear effect 7%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add the alignment-enhanced prior path loss L_AEP | PHOENIX14T · BLEU-4 | 4.74 -> 29.10 (Dev); 5.34 -> 29.27 (Test) | Helped | Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · 21 citations) |
| add language descriptions | Average (540) · PSNR | 24.31 -> 25.72 | Helped | Language-guided Image Reflection Separation (CVPR 2024 · 21 citations) |
| Add reverse Kullback-Leibler divergence loss | Short subset · average score | 62.4 to 63.0 | Helped | Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Large Model Enhancement (CVPR 2025 · 11 citations) |
| add SSS | QaTa-COV19, 50% labeled · Dice (%) | 81.10 -> 82.52 | Helped | DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation (AAAI 2025 · 10 citations) |
| Use ProtAnno-D with property-driven sampling instead of pretrain+finetune | DeepLoc Sub · accuracy | 74.98 -> 75.77 | Helped | ProtCLIP: Function-Informed Protein Multi-Modal Learning (AAAI 2025 · 10 citations) |
| Add hierarchical LLM knowledge augmentation | Zero-shot phase recognition across Cholec80, AutoLaparo, StrasBypass70, and BernBypass70 · accuracy | 41.7 -> 42.4; 23.3 -> 24.9; 26.9 -> 32.3; 22.8 -> 23.8 | Helped | Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation (NeurIPS 2024 · 10 citations) |
| Add FCA | MixATIS · Overall accuracy | 52.4 -> 52.8 | Helped | Aligner²: Enhancing Joint Multiple Intent Detection and Slot Filling via Adjustive and Forced Cross-Task Alignment (AAAI 2024 · 9 citations) |
| Add BEV-GIE | Nr3D · accuracy (%) | 65.0 -> 66.2 | Helped | Cross-Modal Match for Language Conditioned 3D Object Grounding (AAAI 2024 · 6 citations) |
| Use SaCo loss without pseudo-affinity mimicking instead of the full scratch-training method | ImageNet-1K · Top-1 accuracy | 18.3 -> 17.5 | Hurt | SaCo Loss: Sample-wise Affinity Consistency for Vision-Language Pre-training (CVPR 2024 · 5 citations) |
| add difference-aware cross-modal learning | CLEVR-Change · CIDEr-D | 131.8 -> 133.7 | Helped | DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning (AAAI 2025 · 3 citations) |
| Add PTA to text-guided image mixing | VQA dev (4M) · accuracy | 77.89 -> 78.63 | Helped | TiMix: Text-Aware Image Mixing for Effective Vision-Language Pre-training (AAAI 2024 · 3 citations) |
| Add each-subspace discriminability to all-subspace discriminability | Flickr30K, Swin-224 + BERT · Image-to-Text R@1 | 80.6 -> 83.7 | Helped | DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 29 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO · 4 of 29 papers
- Flickr30K · 3 of 29 papers
- MS-COCO · 3 of 29 papers
- SBU · 2 of 29 papers
- TextVQA · 2 of 29 papers
- A-OKVQA · 2 of 29 papers
- CC3M · 2 of 29 papers
Baselines
- CLIP · 5 of 29 papers
- E5-V · 3 of 29 papers
- SigLIP · 3 of 29 papers
- SigLip-so400M · 2 of 29 papers
- OpenCLIP · 2 of 29 papers
- TIES-merging · 2 of 29 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 91 papers; from OpenAlex and the papers' affiliations).
- Peking University · 5 papers
- University of Science and Technology of China · 5 papers
- Tsinghua University · 4 papers
- Alibaba Group · 4 papers
- Zhejiang University · 4 papers
- Fudan University · 4 papers
- Huawei · 3 papers
- Xiamen University · 3 papers
- Shandong University · 2 papers
- Amazon · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Unimodal-to-multimodal representation transfer (145 papers)
- Probabilistic embeddings for cross-modal retrieval (108 papers)
- Cross-modal hashing (45 papers)
Elsewhere in the topic
- Multimodal fusion with missing modalities (208 papers)
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Multimodal sentiment analysis (106 papers)
- Audio-visual speech separation (76 papers)
- Audio-visual source localization (67 papers)
- Self-supervised audio-visual representation learning (44 papers)
- Multi-modal object re-identification (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).