Research map / Vision-language and multimodal / CLIP and zero-shot
TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TripletCLIP, which pairs LLM-generated hard negative captions with SDXL-turbo-generated negative images (the 13M TripletData) and trains CLIP with a triplet contrastive loss, improving SugarCrepe compositionality on CC3M and CC12M at equal compute.
Problem
CLIP models show weak compositional reasoning ("bag of words" behavior) because random batch sampling rarely exposes them to hard negatives, and prior hard-negative augmentation has been text-only with rule-based, often unrealistic captions.
Approach
Generate hard negative captions with an LLM via in-context prompting, synthesize matching negative images with a text-to-image diffusion model, and train CLIP with a triplet contrastive loss that alternates the roles of positive and negative image-text pairs so that both modalities contribute hard negatives.
Key results
- TripletCLIP on CC12M reaches 74.45 SugarCrepe average vs 63.54 for LaCLIP (+10.91%), and on CC3M reaches 63.49 vs 54.09 (+9.40%).
- TripletCLIP improves zero-shot ImageNet1k top-1 on CC12M from 19.72 to 23.31 and on CC3M from 3.79 to 7.32.
- Negative images alone (NegImage) give only 56.28 SugarCrepe on CC3M, while combining negative captions and images in TripletCLIP gives 63.49.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with hard negative image-text pairs and triplet contrastive loss instead of LaCLIP | ViT-B/32 · SugarCrepe (CC3M and CC12M) | overall accuracy: 54.09 -> 63.49; 63.54 -> 74.45 | Helped |
| Use both negative captions and negative images rather than negative captions alone (ablation) | TripletCLIP · SugarCrepe (CC3M) | accuracy: 61.69 -> 63.49 | Helped |
| Use negative images without negative captions (ablation) | LaCLIP · SugarCrepe (CC3M) | accuracy: 54.09 -> 56.28 | Helped |
| Use negative captions without negative images (ablation) | LaCLIP · SugarCrepe (CC3M) | accuracy: 54.09 -> 61.69 | Helped |
| Filter TripletData using CLIP Score (ablation) | TripletCLIP · SugarCrepe (CC3M) | accuracy: 63.49 -> 66.09 | Helped |
| Freeze the vision encoder while training the text encoder (ablation) | TripletCLIP · SugarCrepe | accuracy: 0.6373 -> 0.6227 | Hurt |
| Train the vision encoder while freezing the text encoder (ablation) | TripletCLIP · SugarCrepe | accuracy: 0.5886 -> 0.6923 | Helped |
| Generate negative captions with Gemma-2b-it rather than Mistral-7b-instruct-v0.2 (ablation) | NegCLIP++ · SugarCrepe (CC3M) | accuracy: 61.69 -> 56.00 | Hurt |
| Add TripletLoss to CyCLIP (ablation) | CyCLIP · SugarCrepe (CC3M) | accuracy: 54.62 -> 58.64 | Helped |
Surprising findings
- Negative images alone barely improve compositionality, but they are essential as a regularizer when paired with negative captions.
- LaCLIP outperforms TripletCLIP on Winoground, contrary to gains on other compositional benchmarks.
Limitations
- Experiments limited to ViT-B/32 and small datasets (CC3M/CC12M) due to academic compute; no scaling to hundreds of millions of pairs.
- Generated negative images are imperfect (about 76% of question-answer checks pass for SDXL-turbo), and retrieval/classification gains on some tasks are small.
- Performance drops on Winoground and on finetuning-based retrieval relative to some baselines.
Experimental setup
Datasets: CC3M CC12M TripletData
Benchmarks: SugarCrepe Winoground Valse Cola Flickr30k MSCOCO ImageNet1k VTAB CLIP-Benchmark datasets
Metrics: accuracy recall@k winoground text/image/group score
Baselines: LaCLIP LaCLIP + HN NegCLIP NegCLIP++ CLIP CyCLIP DAC TSVLC CoN-CLIP DataComp models
Builds on: CLIP NegCLIP LaCLIP LiT SDXL-turbo
Closest papers by meaning
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.91)
- Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.90)
- AmorLIP: Efficient Language-Image Pretraining via Amortization (NeurIPS 2025 · similarity 0.90)
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.90)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.90)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.