Research map / Vision-language and multimodal / Cross-modal learning
Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Geometrical Structure Consistency (GSC), which uses cross-modal and intra-modal similarity structures to estimate soft true-correspondence labels for filtering noisy image-text pairs during retrieval training.
Problem
Noisy correspondence (mismatched cross-modal pairs) in human-annotated or web-crawled image-text datasets degrades cross-modal retrieval; prior methods treat it as uni-modal noisy label learning and ignore how it distorts cross-modal and intra-modal structures.
Approach
Well-established cross-modal and intra-modal similarity structures early in training (via memorization effect) separate clean from noisy pairs; GSC computes soft correspondence indicators from these structural differences and uses them to purify both the cross-modal contrastive loss and the intra-modal structure loss, in a single-model framework with temporal ensembling and optional dual networks.
Key results
- On Flickr30K with 40% simulated noise, GSC-SGR reaches a recall sum of 497.3 versus 491.1 for NCR-SGR and 487.5 for BiCro-SGRAF.
- On CC152K (real-world noise), GSC-SGR achieves a recall sum of 375.1, above BiCro-SGRAF at 370.2.
- On MS-COCO 5K with 50% noise, GSC (416.3 sum in the CLIP comparison table) beats finetuned CLIP-B (131.1 sum) and NCR (404.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply GSC to SGR | SGR · Flickr30K, 40% noise | Sum: 487.5 -> 497.3 | Helped |
| apply GSC to SGR | SGR · MS-COCO, 60% noise | Sum: 505.5 -> 511.7 | Helped |
| apply GSC | cross-modal retrieval · NUS-WIDE, 40% noise | Sum: 326.0 -> 342.1 | Helped |
| apply GSC to SGR | SGR · CC152K | Sum: 370.2 -> 375.1 | Helped |
| apply GSC | MS-COCO 5K fine-tuning · MS-COCO 5K, 50% noise | Sum: 404.6 -> 416.3 | Helped |
| add the intra-modal loss (ablation) | GSC on SGR · Flickr30K, 40% noise | Sum: 489.0 -> 497.3 | Helped |
| use dual networks instead of a single network (ablation) | GSC on SGR · Flickr30K, 40% noise | Sum: 482.9 -> 497.3 | Helped |
| add temporal ensembling momentum (ablation) | GSC on SGR · Flickr30K, 40% noise | Sum: 491.1 -> 497.3 | Helped |
| increase batch size from 32 to 128 (ablation) | GSC on SGR · Flickr30K, 40% noise | Sum: 483.9 -> 497.3 | Helped |
| increase batch size from 128 to 192 (ablation) | GSC on SGR · Flickr30K, 40% noise | Sum: 497.3 -> 497.8 | Helped |
Surprising findings
- Fine-tuning CLIP-B on MS-COCO with 50% noise collapses its recall sum to 131.1, far below its zero-shot performance, showing large-model fine-tuning is fragile under noisy correspondence.
Limitations
- Relies on the memorization effect in early training to establish well-formed structures, which may not hold for all data.
- Evaluated mainly with simulated noise on Flickr30K and MS-COCO; real-world noise evaluated only on CC152K.
Experimental setup
Datasets: Flickr30K MS-COCO NUS-WIDE CC152K
Benchmarks: Flickr30K MS-COCO NUS-WIDE CC152K
Metrics: R@1 R@5 R@10 recall sum
Baselines: SGR SGRAF NCR DECL RINCE MSCN BiCro CLIP
Builds on: NCR SGRAF DivideMix BiCro memorization effect
Closest papers by meaning
- Robust Noisy Correspondence Learning with Equivariant Similarity Consistency (CVPR 2024 · similarity 0.88)
- Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.87)
- ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning (CVPR 2025 · similarity 0.87)
- Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation (AAAI 2024 · similarity 0.86)
- Self-Supervised Spatial Correspondence Across Modalities (CVPR 2025 · similarity 0.84)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.