Research map / Vision-language and multimodal / CLIP and zero-shot
Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CE-CLIP, which fine-tunes CLIP (and X-VLM) with intra-modal contrast and an adaptive-threshold cross-modal ranking loss over automatically generated POS-based hard negatives, improving visio-linguistic compositional reasoning without extra parameters or annotations.
Problem
Vision-language models such as CLIP have weak compositional reasoning (relations, attributes, actions, objects), behaving like bag-of-words, because standard image-text contrastive learning does not exploit hard negative captions well.
Approach
Generate four types of hard negative captions (relation, attribute, action, object) via spaCy POS parsing and RoBERTa infilling, then add an intra-modal contrastive loss among positive and hard-negative texts and a cross-modal ranking hinge loss with per-type adaptive thresholds that grow as training progresses (curriculum-like).
Key results
- CE-CLIP improves over CLIP by 23.7% on ARO-Relation and 13.5% on ARO-Attribute, and by 12.1% on SugarCrepe.
- CE-CLIP+ (trained on COCO+CC3M+VG) reaches 83.6 on ARO-Relation and 77.1 on ARO-Attribute, and 87.5% average on SugarCrepe.
- CE-CLIP drops slightly on zero-shot ImageNet1K (-0.5) and CIFAR10/100 classification, while T2I R@5 improves by 13.4 for CE-CLIP+.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train CLIP with intra-modal contrast and cross-modal rank on featured hard negatives | CLIP · ARO Relation | accuracy: 59.3 -> 83.6 | Helped |
| Train CLIP with intra-modal contrast and cross-modal rank on featured hard negatives | CLIP · ARO Attribute | accuracy: 62.9 -> 77.1 | Helped |
| Train CLIP with intra-modal contrast and cross-modal rank on featured hard negatives | CLIP · VALSE Avg. | accuracy: 65.3 -> 76.7 | Helped |
| Train CLIP with intra-modal contrast and cross-modal rank on featured hard negatives | CLIP · VL-CheckList Avg. | accuracy: 69.2 -> 78.4 | Helped |
| Train CLIP with intra-modal contrast and cross-modal rank on featured hard negatives | CLIP · SugarCrepe REPLACE Avg | accuracy: 85.0 -> 89.3 | Helped |
| Add intra-modal contrast to hard-negative image-text contrastive training (ablation) | CE-CLIP+ · ARO Relation | accuracy: 81.6 -> 82.6 | Helped |
| Add cross-modal rank to hard-negative image-text contrastive training (ablation) | CE-CLIP+ · ARO Relation | accuracy: 81.6 -> 82.3 | Helped |
| Combine intra-modal contrast and cross-modal rank (ablation) | CE-CLIP+ · ARO Relation | accuracy: 82.6 -> 83.6 | Helped |
| Use adaptive rather than fixed thresholds for cross-modal rank (ablation) | CE-CLIP+ · Five compositional benchmarks | accuracy | Helped |
| Set the cross-modal rank threshold upper bound to 10 (ablation) | CE-CLIP+ · Compositional benchmarks | accuracy | Helped |
| Add object, attribute, relation, and action hard-negative types (ablation) | CE-CLIP+ · Compositional benchmarks | accuracy | Helped |
| Increase training data from COCO to COCO, CC3M, and Visual Genome (ablation) | CE-CLIP · ARO Relation | accuracy: 83 -> 83.6 | Helped |
Surprising findings
- Scaling CLIP to ViT-G/14 trained on LAION-2B does not surpass base-sized CE-CLIP on VL-CheckList, suggesting scale alone does not solve compositionality.
- Relation hard negatives' similarity gap escalates rapidly without an upper bound on the threshold, requiring a cap of 10 for stable training.
Limitations
- Relies exclusively on textual hard negatives, which may explain lower image-to-text retrieval gains.
- Out-of-distribution weakness on VL-CheckList when trained only on COCO.
- Relation hard negatives are implausible sentences that need an upper bound on the adaptive threshold for stability.
Experimental setup
Datasets: COCO CC3M Visual Genome
Benchmarks: ARO VALSE VL-CheckList SugarCrepe COCO image-text retrieval ImageNet1K CIFAR10 CIFAR100 VQAv2
Metrics: accuracy recall@5
Baselines: NegCLIP CLIP-SVLC CLIP CyCLIP X-VLM DAC syn-CLIP CLIP-SGVL BLIP BLIP2 BEIT3 MiniGPT-4
Builds on: CLIP NegCLIP X-VLM hinge loss SLVC
Closest papers by meaning
- Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.92)
- ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs (NeurIPS 2024 · similarity 0.92)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.91)
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.90)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.90)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.