Research map / Vision-language and multimodal / Cross-modal learning
Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CUSA, a plug-and-play soft-label alignment framework that uses uni-modal teacher models (Unicom for images, all-mpnet-base-v2 for text) to add cross-modal (CSA) and uni-modal (USA) KL-divergence losses to image-text retrieval models.
Problem
Image-text retrieval suffers from inter-modal matching missing (false negatives in contrastive batches) and intra-modal semantic loss (poor recognition of similar samples within each modality).
Approach
Use soft-label similarity distributions from frozen uni-modal pre-trained encoders as targets: CSA aligns cross-modal logits to teacher intra-modal similarities to overcome false negatives, and USA adds an intra-modal alignment loss through an extra projection layer so similar samples cluster within each modality.
Key results
- On MSCOCO 5K, CUSA raises SGRAF RSUM from 429.7 to 439.2 and CLIP ViT-B/32 RSUM from 422.6 to 429.7.
- On ECCV Caption, CUSA improves CLIP ViT-B/32 text-to-image mAP@R from 41.7 to 45.2.
- Ablation on CLIP ViT-B/32 shows USA raises uni-modal image retrieval average from 40.7 to 47.8, while CSA alone gives the larger cross-modal RSUM gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CUSA to SGRAF | SGRAF · MSCOCO 5K test set | RSUM: 429.7 -> 439.2 | Helped |
| Add CUSA to SGRAF | SGRAF · Flickr30K 1K test set | RSUM: 500.8 -> 514.1 | Helped |
| Add CUSA to CLIP ViT-B/32 | CLIP ViT-B/32 · MSCOCO 5K test set | RSUM: 422.6 -> 429.7 | Helped |
| Add CUSA to CLIP ViT-B/32 | CLIP ViT-B/32 · Flickr30K 1K test set | RSUM: 520.0 -> 526.3 | Helped |
| Add CUSA to SGRAF | SGRAF · ECCV Caption | Image-to-text mAP@R: 28.1 -> 29.5 | Helped |
| Add CUSA to CLIP ViT-B/32 | CLIP ViT-B/32 · CUB | R@1: 41.5 -> 49.6 | Helped |
| Add CUSA to X2VLM base | X2VLM base · STS tasks average | Spearman's correlation: 29.4 -> 51.2 | Helped |
| Add CSA alignment (ablation) | CLIP ViT-B/32 + USA · MSCOCO | RSUM: 425.7 -> 429.7 | Helped |
| Add CSA alignment (ablation) | CLIP ViT-B/32 + USA · Flickr30K | RSUM: 523.4 -> 526.3 | Helped |
| Add USA alignment (ablation) | CLIP ViT-B/32 + CSA · Image retrieval average | Avg. IR: 40.4 -> 46.5 | Helped |
| Add USA alignment (ablation) | CLIP ViT-B/32 + CSA · STS tasks average | Avg. STS: 66.5 -> 73.2 | Helped |
| Add USA alignment (ablation) | CLIP ViT-B/32 + CSA · MSCOCO | RSUM: 427.7 -> 429.7 | Helped |
Surprising findings
- CSA alone reduces uni-modal STS average (69.4 to 66.5) for CLIP ViT-B/32, while adding USA recovers and improves it.
Limitations
- Teacher choice (Unicom, all-mpnet-base-v2) is not ablated; the claim that any teacher works is untested in the paper.
- Gains on some metrics are small or mixed (e.g., CLIP ViT-L/14 STS-R decreases and CSA alone lowers uni-modal STS).
Experimental setup
Datasets: MSCOCO Flickr30K ECCV Caption CUB SOP In-Shop iNaturalist STS 2012-2016 STS Benchmark SICK-Relatedness
Benchmarks: MSCOCO 5K Flickr30K 1K ECCV Caption CUB SOP In-Shop iNaturalist STS 2012-2016 STS Benchmark SICK-R
Metrics: R@1 R@5 R@10 RSUM mAP@R R-P Spearman correlation
Baselines: SCAN VSE∞ VSRN++ NAAF SGR SAF SGRAF CLIP BLIP OmniVL X2VLM
Builds on: InfoNCE TCL SGRAF CLIP X2VLM
Closest papers by meaning
- Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network (CVPR 2023 · similarity 0.89)
- DiCA: Disambiguated Contrastive Alignment for Cross-Modal Retrieval with Partial Labels (AAAI 2025 · similarity 0.89)
- Multi-Level Cross-Modal Alignment for Image Clustering (AAAI 2024 · similarity 0.88)
- RoDA: Robust Domain Alignment for Cross-Domain Retrieval Against Label Noise (AAAI 2025 · similarity 0.88)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.86)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.