Research map / Vision-language and multimodal / Cross-modal learning
How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contrastive Partial Ranking Distillation (CPRD), which distills hard-negative ranking from a cross-encoder into a dual-encoder via contrastive learning, ignoring order among easy and invalid negatives, for efficient image-text retrieval.
Problem
Dual-encoders for image-text retrieval are efficient but less accurate than cross-encoders; naive logit distillation from cross-encoder to dual-encoder is ineffective because their similarity score distributions differ substantially.
Approach
Distill only the relative order among valid hard negatives (those the cross-encoder scores above a threshold) using an InfoNCE-style contrastive partial ranking loss, so the distillation objective coordinates with the dual-encoder's contrastive training and avoids matching score distributions.
Key results
- On MSCOCO 5K fine-tuned, the dual-encoder with CPRD reaches 69.7 image-to-text R@1 vs 66.7 for the baseline dual-encoder (+3.0), and R@sum 474.4 vs 460.3.
- Compared with COTS (5.3M pairs), CPRD dual-encoder at 5.1M pairs gains 1.8% and 1.6% R@1 on MSCOCO fine-tuning per the paper.
- Ablation on valid-negative threshold m shows m=0.75 is best (R@sum 313.2), while m=0 degrades below the baseline (290.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill cross-encoder ranking knowledge with CPRD | Dual-encoder · Flickr30K (1K test set) | R@S: 550.7 -> 556.0 | Helped |
| Distill cross-encoder ranking knowledge with CPRD | Dual-encoder · MSCOCO (5K test set) | R@S: 460.3 -> 474.4 | Helped |
| Distill cross-encoder ranking knowledge with CPRD | Dual-encoder · MSCOCO (5K) zero-shot | R@S: 432.5 -> 444.7 | Helped |
| Distill cross-modal ranking knowledge with CPRD | Dual-encoder · CrissCrossed Captions | SITS Spearman's R ×100: 61.8±1.4 -> 64.3±1.3 | Helped |
| Use KL-divergence score-distribution distillation instead of no distillation | Dual-encoder · MSCOCO (5K) fine-tuned | R@S: 297.8 -> 303.3 | Helped |
| Use M3SE margin distillation instead of no distillation | Dual-encoder · MSCOCO (5K) fine-tuned | R@S: 297.8 -> 290.4 | Hurt |
| Use normalized M3SE (R-M3SE) instead of no distillation | Dual-encoder · MSCOCO (5K) fine-tuned | R@S: 297.8 -> 302.3 | Helped |
| Adapt the valid-negative threshold so only one hard negative is valid (ablation) | CPRD · MSCOCO (5K) fine-tuned | R@S: 313.2 -> 307.7 | Hurt |
| Increase cross-encoder queue size from 0 to 16384 (ablation) | CPRD · MSCOCO (5K) fine-tuned | R@S: 299.9 -> 313.2 | Helped |
| Increase considered hard negatives from K=2 to K=16 (ablation) | CPRD · MSCOCO (5K) fine-tuned | R@S: 300.5 -> 312.4 | Helped |
| Set valid hard-negative threshold m to 0 instead of 0.75 (ablation) | CPRD · MSCOCO (5K) fine-tuned | R@S: 313.2 -> 290.4 | Hurt |
| Set valid hard-negative threshold m to 1.0 instead of 0.75 (ablation) | CPRD · MSCOCO (5K) fine-tuned | R@S: 313.2 -> 297.8 | Hurt |
Surprising findings
- Vanilla KL logit distillation gives only limited gains, and M3SE even hurts performance below the no-distillation baseline due to differing similarity ranges.
- Distilling the order among all negatives (threshold m=0) performs worse than the baseline, showing that learning order among easy negatives is harmful.
Limitations
- Online cross-encoder scoring adds training cost; offline approach needs precomputed top-N negatives.
- Threshold m and K are hyperparameters tuned on MSCOCO-based ablations.
Experimental setup
Datasets: CC4M CC3M MSCOCO Flickr30K CrissCrossed Captions
Benchmarks: MSCOCO Flickr30K CrissCrossed Captions
Metrics: R@1 R@5 R@10 R@sum Spearman's R bootstrap correlation
Baselines: KL logit distillation M3SE R-M3SE COTS VSE∞ COOKIE LightningDOT CLIP ALIGN VinVL-base ALBEF BLIP
Builds on: ALBEF InfoNCE Margin-MSE M3SE MoCo-style momentum queues
Closest papers by meaning
- Lightweight Contrastive Distilled Hashing for Online Cross-modal Retrieval (AAAI 2025 · similarity 0.85)
- DiCA: Disambiguated Contrastive Alignment for Cross-Modal Retrieval with Partial Labels (AAAI 2025 · similarity 0.85)
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.85)
- Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024 · similarity 0.85)
- Retrieval Across Any Domains via Large-scale Pre-trained Model (ICML 2024 · similarity 0.85)
- Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.