Research map / Vision-language and multimodal / Cross-modal learning
Generalized Contrastive Learning for Universal Multimodal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Generalized Contrastive Learning (GCL), a loss that contrasts text, image, and fused image-text embeddings across all modality pairs in a mini-batch, fine-tuning VISTA, CLIP-SF and TinyCLIP-SF on off-the-shelf image-caption pairs (LCS-558K) to improve universal multimodal retrieval.
Problem
Multimodal retrieval over databases mixing text-only, image-only, and fused image-text keys suffers from a modality gap; existing fixes rely on costly curated triplet datasets that fail to generalize to unseen modality combinations.
Approach
Extend standard contrastive learning to all modality pairs (image, text, fused image-text) within a mini-batch of ordinary image-caption pairs, masking same-modality positives, so the model learns retrieval across all modality combinations without new triplet datasets.
Key results
- On M-BEIR global setting, GCL raises VISTA average Recall@50 from 21.18 (pretrained) to 34.06, versus 25.28 for triplet-based CL.
- On M-BEIR global setting, CLIP-SF average improves from 14.92 to 21.89 with GCL; TinyCLIP-SF improves from 17.36 to 22.71.
- On MMEB Recall@1, VISTA average rises from 40.3 to 44.9 with GCL, and CLIP-SF from 39.3 to 44.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply GCL instead of standard contrastive learning to image-text pairs | VISTA · M-BEIR global | Recall@50 avg.: 24.65 -> 34.06 | Helped |
| Apply GCL instead of standard contrastive learning to image-text pairs | CLIP-SF · M-BEIR global | Recall@50 avg.: 17.52 -> 21.89 | Helped |
| Apply GCL instead of standard contrastive learning to image-text pairs | VISTA · M-BEIR local | Recall@5/Recall@10 avg.: 31.49 -> 35.56 | Helped |
| Apply GCL instead of standard contrastive learning to image-text pairs | CLIP-SF · M-BEIR local | Recall@5/Recall@10 avg.: 29.51 -> 30.42 | Helped |
| Apply GCL instead of standard contrastive learning to image-text pairs | VISTA · MMEB | Recall@1 avg.: 43.4 -> 44.9 | Helped |
| Apply GCL instead of standard contrastive learning to image-text pairs | CLIP-SF · MMEB | Recall@1 avg.: 29.9 -> 44.2 | Helped |
| Apply GCL to image-text pairs | VISTA · CoVR | Recall@1: 33.76 -> 37.52 | Helped |
| Apply GCL to image-text pairs | CLIP-SF · CoVR | Recall@1: 19.68 -> 37.60 | Helped |
| Apply GCL | TinyCLIP-SF · M-BEIR global | average score: 17.36 -> 22.71 | Helped |
| Add intra-modality separation loss to standard contrastive learning (ablation) | VISTA · M-BEIR global | Recall@50 avg.: 24.65 -> 30.45 | Helped |
| Add cross-modal alignment loss terms L_i2t and L_t2i (ablation) | GCL on VISTA · M-BEIR global, q_t → c_i, VisualNews | Recall@50: 2.91 -> 16.64 | Helped |
| Add fused-candidate learning loss terms L_i2it and L_t2it (ablation) | GCL on VISTA · M-BEIR global, q_t → (c_i, c_t), WebQA | Recall@50: 52.93 -> 64.00 | Helped |
Surprising findings
- Fine-tuning VISTA on triplet-based composed datasets degrades cross-modal tasks relative to the pretrained model.
- CLIP-SF performance drops sharply from local to global setting, indicating summed fused embeddings limit unified space construction.
Limitations
- Code is under internal legal review and not yet released.
- Gains on scenario-specific tasks such as CIRR and FashionIQ are sometimes slight drops, attributed to general-purpose fine-tuning data.
- Loss weights were not tuned; all components added with identical weights.
Experimental setup
Datasets: LLaVA Visual Instruct Pretrain LCS-558K IT2I triplet dataset (VISTA, baseline) T2IT triplet dataset (VISTA, baseline)
Benchmarks: M-BEIR MMEB CoVR
Metrics: Recall@K Recall@1
Baselines: VISTA pretrained VISTA CL+Triplet CLIP-SF pretrained CL+Pairwise AlignCLIP intra-modality separation TinyCLIP-SF
Builds on: CLIP VISTA UniIR AlignCLIP
Closest papers by meaning
- MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.87)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.87)
- Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.87)
- Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (NeurIPS 2026 · similarity 0.87)
- Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval (CVPR 2025 · similarity 0.87)
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.