Research map / Vision-language and multimodal / Human and embodied modeling
The Hard Positive Truth about Vision-Language Compositionality
Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, Ranjay Krishna
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune CLIP with hard negatives only, evaluated with hard positives added | CLIP ViT-B/32 · REPLACE benchmark | accuracy drop (Orig -> Aug): avg 24.4 point drop vs CLIP's 14.9 | Hurt |
| hard negative finetuning (DAC-LLM) on REPLACE | CLIP ViT-B/32 · REPLACE | Brittleness: 23.2 -> 40.1 | Hurt |
| add hard positives to hard negative finetuning (ablation) | CLIP ViT-B/32 · REPLACE | Augmented test accuracy: 55.7 -> 58.0 | Improved |
| add hard positives to hard negative finetuning (ablation) | CLIP ViT-B/32 · REPLACE | Brittleness: 21.0 -> 16.9 | Improved |
| add hard positives to hard negative finetuning (ablation) | CLIP ViT-B/32 · SWAP | Brittleness: 25.1 -> 22.9 | Improved |
| add generic paraphrase positives (SVLC+Pos) to SVLC | SVLC · REPLACE | Augmented test accuracy: 44.5 -> 45.0 | No clear effect |
| train with hard positives only (hard negative weight 0) (ablation) | CLIP ViT-B/32 · REPLACE | Augmented test accuracy: 57.6 (0.75 HN) -> 49.8 (0 HN) | Hurt |
| increase hard negative loss weight (ablation) | CLIP ViT-B/32 · REPLACE | Brittleness: 15.8 -> 16.6 brittleness; aug acc 49.8 -> 57.6 | Mixed |
| finetune on both SWAP and REPLACE vs replace-only (ablation) | CLIP ViT-B/32 · REPLACE | Augmented test accuracy: 59.0 (replace-only) vs 58.0 (multi-task) | No clear effect |
Closest papers by meaning
- Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models (ICLR 2025 · similarity 0.90)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.90)
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
