Research map / Vision-language and multimodal / Cross-modal learning
Robust Noisy Correspondence Learning with Equivariant Similarity Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Equivariant Similarity Consistency (ESC), a regularizer that uses anchor pairs to separate clean and noisy image-text correspondences and trains with a soft triplet loss in a co-teaching setup.
Problem
Cross-modal image-text matching trained on web data with noisy (mismatched) correspondences; the standard hinge triplet loss is not robust to such noise, which hurts clean/noisy data division and matching performance.
Approach
Semantic variation caused by image changes should be proportional to that caused by text changes for matched pairs; ESC enforces cross-instance similarity equality between an anchor clean pair and other pairs in a mini-batch to flag noisy pairs, and adds a pseudo-negative ESC term during training on clean data.
Key results
- On CC152K, ESC achieves rSum 375.9 vs 373.7 for CRCL (best baseline), with text-to-image R@1 44.8.
- On Flickr30K with 60% synthetic noise, ESC improves rSum by 3.5% over the best baseline (475.0 vs 471.5 for CRCL).
- On MS-COCO 1K with 40% noise, ESC reaches rSum 523.9 vs 522.0 for CRCL? (ESC improves over best baseline by 5.8% per text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add matching regularization LESC to ESC (ablation) | ESC · Flickr30K, 40% noise | R@1 image-to-text: 73.0 -> 76.1 | Helped |
| Add division regularization lESC to ESC (ablation) | ESC · Flickr30K, 40% noise | R@1 image-to-text: 74.4 -> 76.1 | Helped |
| Add warmup to ESC (ablation) | ESC · Flickr30K, 40% noise | R@1 image-to-text: 5.8 -> 76.1 | Helped |
| Add matching regularization LESC to ESC (ablation) | ESC · Flickr30K, 40% noise | R@1 text-to-image: 4.6 -> 56.0 | Helped |
| Add division regularization lESC to ESC (ablation) | ESC · Flickr30K, 40% noise | R@1 text-to-image: 55.6 -> 56.0 | Helped |
| Add warmup to ESC (ablation) | ESC · Flickr30K, 40% noise | R@1 text-to-image: 4.6 -> 56.0 | Helped |
Surprising findings
- The division regularization L_ESC is not backpropagated yet substantially affects performance, per the ablation (removing it drops Flickr30K 40% R@1 from 76.1 to 73.0 for image-to-text).
Limitations
- Evaluated only on image-text matching; generalization to other cross-modal tasks is not shown.
- Performance degrades at severe noise and the paper dismisses 80% noise as impractical.
- Relies on the memorization effect and a warmup phase plus hyperparameters (alpha1, alpha2, beta, m).
Experimental setup
Datasets: Flickr30K MS-COCO Conceptual Captions (CC152K)
Benchmarks: Flickr30K MS-COCO 1K MS-COCO 5K CC152K
Metrics: R@1 R@5 R@10 rSum
Baselines: SCAN VSRN IMRAM SAF SGR NCR DECL BiCro MSCN CRCL
Builds on: NCR SGRAF triplet loss (VSE++) co-teaching BiCro
Closest papers by meaning
- Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.91)
- Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning (CVPR 2024 · similarity 0.88)
- Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation (AAAI 2024 · similarity 0.88)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.88)
- Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024 · similarity 0.87)
- ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.