Research map / Vision-language and multimodal / Cross-modal learning
Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L2RM, an optimal-transport framework that rematches partially mismatched image-text pairs using a self-supervised learnable cost function and a masked partial OT plan, improving robustness of cross-modal retrieval models.
Problem
Cross-modal retrieval models trained on web-collected image-text data suffer from partially mismatched pairs (PMPs), and prior methods only down-weight these pairs, underusing them.
Approach
Instead of discarding or down-weighting mismatched pairs, rematch them via a partial optimal transport problem with positives masked, where the transport cost is learned self-supervisedly from reconstructed pairs with known matching matrices, and the resulting refined alignment supervises training via symmetric KL divergence.
Key results
- On Flickr30K at 0.6 mismatching rate, L2RM-SGRAF reaches rSum 467.6 versus 456.6 for the best baseline RCL-SGRAF.
- On CC152K with real-world PMPs, L2RM-SGRAF achieves rSum 374.2 versus 364.4 for RCL-SGRAF and 364.3 for DECL-SGRAF.
- Ablation on Flickr30K (0.6 MRate) shows the learnable cost function gives about +11.9 rSum over a cosine-distance cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply L2RM rematching to SGRAF | SGRAF · Flickr30K, MRate 0.8 | rSum: 394.4 -> 404.0 | Helped |
| Apply L2RM rematching to SGRAF | SGRAF · MS-COCO, MRate 0.8 | rSum: 475.9 -> 482.6 | Helped |
| Apply L2RM rematching to SGRAF | SGRAF · CC152K | rSum: 364.4 -> 374.2 | Helped |
| Add partial OT (ablation) | L2RM-SGRAF · Flickr30K, MRate 0.6 | Image-to-Text R@1: 62.2 -> 65.1 | Helped |
| Add positive-pair masking (ablation) | L2RM-SGRAF · Flickr30K, MRate 0.6 | Image-to-Text R@1: 64.9 -> 65.1 | Helped |
| Use the learnable cost function instead of cosine-distance cost (ablation) | L2RM-SGRAF · Flickr30K, MRate 0.6 | Image-to-Text R@1: 61.3 -> 65.1 | Helped |
| Use the symmetric KL-divergence rematching loss instead of KL-divergence (ablation) | L2RM-SGRAF · Flickr30K, MRate 0.6 | Image-to-Text R@1: 64.7 -> 65.1 | Helped |
| Use the symmetric KL-divergence rematching loss instead of InfoNCE loss (ablation) | L2RM-SGRAF · Flickr30K, MRate 0.6 | Image-to-Text R@1: 64.9 -> 65.1 | Helped |
Surprising findings
- Triplet-loss warm-up fails under extreme mismatching (0.8 MRate), with near-zero recall on Flickr30K, whereas InfoNCE warm-up works.
Limitations
- Robustness gains are most evident at high mismatching rates; gains at low rates are smaller.
- Performance tends to decrease as the partial transport mass rho increases, so the transport-mass hyperparameter needs tuning.
Experimental setup
Datasets: Flickr30K MS-COCO CC152K
Benchmarks: Flickr30K MS-COCO CC152K
Metrics: Recall@1 Recall@5 Recall@10 rSum
Baselines: IMRAM SGR SAF SGRAF NCR DECL BiCro RCL
Builds on: NCR BiCro RCL DECL SGRAF Sinkhorn-Knopp
Closest papers by meaning
- Noisy Correspondence Learning With Meta Similarity Correction (CVPR 2023 · similarity 0.90)
- Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.88)
- Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation (AAAI 2024 · similarity 0.88)
- CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer (AAAI 2024 · similarity 0.87)
- Robust Noisy Correspondence Learning with Equivariant Similarity Consistency (CVPR 2024 · similarity 0.87)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.