Research map / Vision-language and multimodal / Cross-modal learning
CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CL2CM, a dual-stream cross-lingual cross-modal retrieval framework that distills multi-level (sentence and self-supervised OT-based word) cross-lingual alignment knowledge from an mBERT-based cross-lingual network into a CLIP-based cross-modal network, evaluated on Multi30K, MSCOCO and VATEX.
Problem
Cross-lingual cross-modal retrieval without annotated vision-target language pairs suffers from the large vision-text heterogeneity gap and noisy machine-translated target captions, and existing methods trade off alignment quality against inference efficiency.
Approach
Transfer cross-lingual knowledge (which has a smaller modality gap) from a multi-level-aligned cross-lingual text network to a dual-stream cross-modal network via relational knowledge distillation, using an OT-based self-supervised word alignment to handle noisy translations, while keeping inference as a cheap dual encoder.
Key results
- On Multi30K en2de, CL2CM reaches 498.0 sumR versus 480.6 for NRCCR among two-stream methods; the XLM-R variant CL2CM† reaches 530.4.
- On VATEX text-to-video, CL2CM achieves R@1 32.1 and SumR 386.9 versus 364.8 for NRCCR.
- CL2CM† is reported to be about 11x faster than the single-stream CCLM at inference, with comparable performance (e.g., 526.3 vs 536.4 en2cs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CL2CM instead of the strongest listed baseline | Cross-lingual image-text retrieval · Multi30K en2de | SumR: 540.0 -> 498.0 | Hurt |
| use CL2CM instead of the strongest listed baseline | Cross-lingual image-text retrieval · Multi30K en2fr | SumR: 545.4 -> 499.7 | Hurt |
| use CL2CM instead of the strongest listed baseline | Cross-lingual image-text retrieval · Multi30K en2cs | SumR: 536.4 -> 485.3 | Hurt |
| use CL2CM instead of the strongest listed baseline | Cross-lingual image-text retrieval · MSCOCO en2zh | SumR: 546.0 -> 522.0 | Hurt |
| use CL2CM instead of the strongest listed baseline | Cross-lingual image-text retrieval · MSCOCO en2ja | SumR: 532.8 -> 515.9 | Hurt |
| use CL2CM instead of NRCCR | Video-text retrieval · VATEX | SumR: 364.8 -> 386.9 | Helped |
| add multi-level alignment (instance-level plus word-level) (ablation) | CL2CM · Multi30K | SumR: 473.2 -> 490.0; 482.3 -> 493.0; 472.6 -> 480.4 | Helped |
| add cross-lingual knowledge transfer (ablation) | CL2CM · Multi30K | SumR: 473.2 -> 491.5; 482.3 -> 492.9; 472.6 -> 480.7 | Helped |
| replace OT pseudo-label generation with cross-attention (ablation) | Self-supervised word alignment in CL2CM · Multi30K | SumR: 495.7 vs 498.0; 496.1 vs 499.7; 480.4 vs 485.3 | Hurt |
| replace vanilla OT with unbalanced OT (ablation) | Self-supervised word alignment in CL2CM · Multi30K | SumR: 497.2; 496.8; 484.0 | No clear effect |
| replace vanilla OT with partial OT (ablation) | Self-supervised word alignment in CL2CM · Multi30K | SumR: 497.3; 497.1; 484.5 | No clear effect |
| use two-stage training rather than end-to-end training (ablation) | CL2CM · Multi30K | SumR | No clear effect |
Surprising findings
- Two-stage training (pre-training the cross-lingual network before transfer) did not yield significant gains over end-to-end training, which the authors attribute to strong pre-trained cross-lingual alignment and small datasets.
Limitations
- Uses an MT-translated target language for training, so translation noise remains a factor.
- Two-stage variant with additional parallel corpora was not explored.
- Lags the single-stream CCLM on Multi30K/MSCOCO when using the same backbone (CL2CM vs CCLM).
Experimental setup
Datasets: Multi30K MSCOCO VATEX
Benchmarks: Multi30K MSCOCO VATEX
Metrics: sumR R@1 R@5 R@10 mAP
Baselines: M3P UC2 CCLM MURAL MLA NRCCR MMP
Builds on: Relational Knowledge Distillation Sinkhorn optimal transport CLIP mBERT
Closest papers by meaning
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.93)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.89)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.88)
- Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval (AAAI 2025 · similarity 0.88)
- CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.