Research map / Label-efficient and robust learning / Noisy labels
Negative Pre-aware for Noisy Cross-Modal Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Negative Pre-aware Cross-modal (NPC) matching, which fine-tunes CLIP on noisy image-text pairs by estimating each sample's negative impact via a GMM-selected clean memory bank and reweighting the loss accordingly.
Problem
Robust cross-modal image-text matching when fine-tuning large vision-language models on noisy correspondences, where existing noise-rectify methods become unstable as the noise ratio increases.
Approach
Instead of filtering or correcting noisy samples, estimate each sample's negative impact before learning it by checking how its update changes the loss on related clean memory-bank entries (chosen by GMM with a strict threshold), then down-weight harmful samples and train on memory entries.
Key results
- On Flickr30K at 60% noise, NPC's R@1 is 15.4% higher than BiCro for image-to-text (83.0 vs 67.6).
- NPC reaches 3.61% variance of R@1 across noise ratios, a 52.79% reduction versus CLIP.
- On CC120K with real noise, NPC improves CLIP R@1 from 68.8 to 71.1 (i2t) and 67.8 to 73.0 (t2i).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply NPC instead of CLIP fine-tuning | CLIP ViT-B/32 · Flickr30K, 60% noise, image-to-text | R@1: 66.3 -> 83.0 | Helped |
| Apply NPC instead of CLIP fine-tuning | CLIP ViT-B/32 · MSCOCO 1K, 60% noise, text-to-image | R@1: 49.7 -> 63.1 | Helped |
| Add confidence weight w (ablation) | NPC · Flickr30K, 60% noise, image-to-text | R@1: 78.2 -> 83.0 | Helped |
| Add confidence weight w (ablation) | NPC · Flickr30K, 60% noise, text-to-image | R@1: 59.0 -> 68.1 | Helped |
| Add memory-bank loss L_MB (ablation) | NPC · Flickr30K, 60% noise, image-to-text | R@1: 78.0 -> 83.0 | Helped |
| Add memory-bank loss L_MB (ablation) | NPC · Flickr30K, 60% noise, text-to-image | R@1: 59.1 -> 68.1 | Helped |
| Use threshold tau 0.99 instead of 0.5 for clean-set selection (ablation) | NPC · Flickr30K, 60% noise, image-to-text | R@1: 78.3 -> 83.1 | Helped |
| Use threshold tau 0.99 instead of 0.5 for clean-set selection (ablation) | NPC · Flickr30K, 60% noise, text-to-image | R@1: 59.2 -> 68.1 | Helped |
| Vary clean-selection threshold tau among 0.5, 0.7, and 0.99 (ablation) | NPC · Flickr30K, 0% noise | R@1: 87.2; 87.6; 87.9 | No clear effect |
Surprising findings
- Plain CLIP fine-tuning drops from 82.3% to 66.3% R@1 on Flickr30K i2t as noise rises from 20% to 60%, showing CLIP-based methods are also unstable under noise.
- SAF and NCR collapse to near-zero R@1 at 60% noise on MSCOCO, indicating rectify-based methods can fail completely.
Limitations
- Relies on a GMM-selected strict clean subset at threshold 0.99, whose quality depends on loss separation.
- Memory bank entries are chosen by image/text similarity, and the paper reports results only with the CLIP ViT-B/32 backbone.
Experimental setup
Datasets: MSCOCO Flickr30K Conceptual Captions (CC120K subset)
Benchmarks: MSCOCO 1K MSCOCO 5K Flickr30K CC120K
Metrics: R@1 R@5 R@10 RSUM variance of R@1 across noise ratios
Baselines: SCAN SAF NCR DECL BiCro CLIP VSE∞ PCME PCME++ PAU
Builds on: CLIP NCR Gaussian Mixture Model loss separation
Closest papers by meaning
- Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation (AAAI 2024 · similarity 0.89)
- Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.88)
- Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution (CVPR 2025 · similarity 0.87)
- Robust Noisy Correspondence Learning with Equivariant Similarity Consistency (CVPR 2024 · similarity 0.86)
- ReconBoost: Boosting Can Achieve Modality Reconcilement (ICML 2024 · similarity 0.86)
- Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.