Research map / Vision-language and multimodal / CLIP and zero-shot
FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves CLIP-style vision-language contrastive pre-training by re-assigning false negatives as extra positives, adding BLIP-2 multi-caption batch augmentation, and training with a sigmoid (SigLIP-style) loss.
Problem
Standard image-text contrastive pre-training assumes one positive per image and uses noisy, low-quality, repetitive web captions, so near-duplicate pairs are treated as false negatives and captions are poor.
Approach
Mine extra positives using image-text, image-image and text-text similarities from a pretrained model, augment each image with five beam-search pseudo-captions from BLIP-2 within the same batch, and train with a sigmoid loss with a data-estimated bias that supports a variable number of positives per sample.
Key results
- Zero-shot classification average over 11 datasets (YFCC15M, ViT-B/32) reaches 50.8% vs 44.6% for HiDeCLIP, and ImageNet 51.1%.
- Zero-shot Flickr30k text retrieval R@1 of 85.3% vs 70.5% for ALIP; image retrieval R@1 67.6% vs 48.9%.
- On ImageNet with ResNet-50 pre-trained on CC12M, zero-shot top-1 reaches 47.4% vs 44.4% for CLIP-Rocket.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fix incorrect negatives using similarity-based positive-pair mining (ablation) | FFF vision-language pretraining · ImageNet | Top-1 accuracy: 18.6 -> 21.3; 23.3 -> 26.8; 31.1 -> 32.9 | Helped |
| Add image-text similarity matching (Sit > p1) to the assignment matrix (ablation) | FFF · ImageNet | Top-1 accuracy: 18.6 -> 21.4 | Helped |
| Add image-image similarity matching (Sii > p2) to the assignment matrix (ablation) | FFF · ImageNet | Top-1 accuracy: 18.6 -> 21.3 | Helped |
| Add text-text similarity matching (Stt > p3) to the assignment matrix (ablation) | FFF · ImageNet | Top-1 accuracy: 18.6 -> 18.8 | Helped |
| Use all similarity components in the assignment matrix (ablation) | FFF · ImageNet | Top-1 accuracy: 18.6 -> 22.0; 23.1 -> 26.8 | Helped |
| Add 1 pseudo-caption per image (ablation) | FFF · ImageNet | Top-1 accuracy: 18.6 -> 23.3 | Helped |
| Increase pseudo-captions from 1 to 3 per image (ablation) | FFF · ImageNet | Top-1 accuracy: 23.3 -> 30.2 | Helped |
| Increase pseudo-captions from 3 to 5 per image (ablation) | FFF · ImageNet | Top-1 accuracy: 30.2 -> 31.1 | Helped |
| Use multiple pseudo-captions jointly in the batch instead of randomly sampling 1 of 5 (ablation) | FFF · ImageNet | zero-shot accuracy: 31.4% to 32.9% | Helped |
| Use multiple pseudo-captions jointly rather than randomly sampling 1 of 5 (ablation) | FFF · ImageNet (YFCC-v2 pretraining) | zero-shot accuracy: 44.1% to 51.1% | Helped |
| Replace BLIP-2 with OFA as the image captioner (ablation) | FFF · ImageNet | Top-1 accuracy: 32.9 -> 32.9 | No clear effect |
| Use supervised contrastive loss instead of sigmoid loss (ablation) | FFF · ImageNet | zero-shot accuracy: 21.3% vs 19.0% | Hurt |
Surprising findings
- Randomly sampling one of five pseudo-captions per image performs notably worse than using all five jointly within the batch (YFCC 44.1% vs 51.1%).
- Supervised contrastive loss underperforms the sigmoid loss on CC3M zero-shot ImageNet (19.0% vs 21.3%).
Limitations
- Pseudo-captions from captioners trained on the same web data can still contain hallucinations and biases.
- Similarity thresholds (p1, p2, p3, p1') are set empirically and the bias beta must be estimated per dataset.
Experimental setup
Datasets: YFCC15M-v2 CC3M CC12M Open30M Open70M
Benchmarks: CIFAR-10 CIFAR-100 Food101 Pets Flowers SUN397 Stanford Cars DTD Caltech101 FGVC-Aircraft ImageNet Flickr30k
Metrics: top-1 accuracy R@1 R@5 R@10
Baselines: CLIP SLIP FILIP DeCLIP DeFILIP HiCLIP HiDeCLIP ALIP UniCLIP SoftCLIP CyCLIP CLOOB
Builds on: CLIP SigLIP sigmoid loss BLIP-2 ALIP DeCLIP
Closest papers by meaning
- SaCo Loss: Sample-wise Affinity Consistency for Vision-Language Pre-training (CVPR 2024 · similarity 0.90)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- MAFA: Managing False Negatives for Vision-Language Pre-training (CVPR 2024 · similarity 0.89)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.88)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.88)
- Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.