Research map / Vision-language and multimodal / CLIP and zero-shot
MAFA: Managing False Negatives for Vision-Language Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAFA, which mines missing positive image-text connections among GRIT-sampled hard negatives with a pretrained discriminator and applies label smoothing to the ITC loss, improving vision-language pre-training.
Problem
False negatives in vision-language pre-training caused by the many-to-many correspondence of web-crawled image-text pairs, which are amplified by hard negative sampling (GRIT) and degrade performance.
Approach
Converts hard negatives identified as likely positives by a connection discriminator (Con-D) into positives in ITC, ITM and MLM losses (ECM), and uses label smoothing on ITC under GRIT sampling (S-ITC) to avoid over-penalizing false negatives without extra compute.
Key results
- MAFA on 4M-Noisy reaches COCO IR R@1 61.2 and TR R@1 78.0, improving over GRIT-VLP (59.6 / 76.6).
- MAFA (4M-Clean) reaches NLVR2 test-P 82.16 and COCO TR R@1 79.4, nearly matching ALBEF trained on 14M images on several metrics.
- With BLIP-2, MAFA improves zero-shot VQAv2 from 46.6 to 50.8 and COCO captioning CIDEr from 118.8 to 125.4, while plain GRIT collapses to 65.9 TR R@1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MAFA false-negative management with ECM and S-ITC | GRIT-VLP · COCO image-text retrieval, TR R@1 | R@1: 76.6 -> 78.0 | Helped |
| Add MAFA false-negative management with ECM and S-ITC | GRIT-VLP · COCO image-text retrieval, IR R@1 | R@1: 59.6 -> 61.2 | Helped |
| Add MAFA false-negative management with ECM and S-ITC | GRIT-VLP · NLVR2 dev | accuracy: 81.40 -> 82.52 | Helped |
| Add MAFA false-negative management | BLIP-2 · VQAv2 val | accuracy: 46.6 -> 50.8 | Helped |
| Add MAFA false-negative management | BLIP-2 · COCO image-text retrieval, TR R@1 | R@1: 82.6 -> 83.7 | Helped |
| Add ECM to identify and use false negatives as positives (ablation) | GRIT-VLP · COCO image-text retrieval, TR R@1 | R@1: 76.6 -> 77.4 | Helped |
| Add S-ITC label smoothing (ablation) | GRIT-VLP · COCO image-text retrieval, TR R@1 | R@1: 76.6 -> 77.5 | Helped |
| Combine ECM and S-ITC (ablation) | GRIT-VLP · COCO image-text retrieval, TR R@1 | R@1: 76.6 -> 78.0 | Helped |
| Use ECM to add false negatives as positives rather than eliminate them (ablation) | MAFA with GRIT sampling · COCO image-text retrieval, IR R@1 | R@1: 61.1 -> 61.2 | Helped |
| Apply MAFA under random sampling (ablation) | ALBEF without momentum distillation · COCO image-text retrieval, IR R@1 | R@1: 57.6 -> 57.8 | Helped |
| Apply S-ITC label smoothing under random sampling (ablation) | ALBEF without momentum distillation · COCO image-text retrieval, TR R@1 | R@1: 74.4 -> 73.5 | Hurt |
| Replace S-ITC with momentum distillation (ablation) | GRIT-VLP · COCO image-text retrieval, TR R@1 | R@1: 77.5 -> 76.1 | Hurt |
Surprising findings
- Plain GRIT sampling causes training failure with BLIP-2 (COCO TR R@1 drops to 65.9 from 82.6), attributed to more false negatives.
- S-ITC label smoothing hurts under random sampling but helps substantially under GRIT sampling.
Limitations
- Relies on a separately pre-trained discriminator (Con-D) fine-tuned on MS-COCO, adding extra forward passes.
- Plain GRIT sampling becomes unstable with larger models such as BLIP-2 without MAFA.
Experimental setup
Datasets: MS-COCO Visual Genome Conceptual Captions SBU Captions Flickr30K
Benchmarks: MS-COCO retrieval Flickr30K retrieval NLVR2 VQAv2 GQA OK-VQA COCO captioning
Metrics: R@1 R@5 R@10 accuracy BLEU@4 CIDEr
Baselines: UNITER VILLA OSCAR ALBEF TCL BLIP GRIT-VLP BLIP-2
Builds on: GRIT-VLP ALBEF BLIP BLIP-2
Closest papers by meaning
- FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models (CVPR 2024 · similarity 0.89)
- Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples (AAAI 2025 · similarity 0.88)
- SaCo Loss: Sample-wise Affinity Consistency for Vision-Language Pre-training (CVPR 2024 · similarity 0.88)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.