Research map / Vision-language and multimodal / Multimodal LLMs
A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets
Zexi Jia, Chuanwei Huang, Hongyan Fei, Yeshuang Zhu, Zhiqiang Yuan, Ying Deng, Jiapei Zhang, Jinchao Zhang, Jie Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace contrastive loss with set-based paired loss L_sets (ablation) | CLIP ViT-B/32 · ARO VG-Relation | accuracy: 62.8 -> 82.3 | Improved |
| add stitched real-data images to training (ablation) | CLIP ViT-B/32 · ARO Order (COCO) | accuracy: 76.1 -> 90.8 | Improved |
| add block-diffusion generated counterfactual images (ablation) | CLIP ViT-B/32 · ARO Order (Flickr) | accuracy: 87.8 -> 93.0 | Improved |
| add negative-text loss L_neg (ablation) | CLIP ViT-B/32 · Winoground | Text score: 21.5 -> 32.6 | Improved |
| increase counterfactual set size from 5 to 20 (ablation) | CLIP ViT-B/32 · ARO VG-Relation | accuracy: 84.9 -> 85.7 | Improved |
| combine local image and local text guidance in block-based diffusion (ablation) | counterfactual generation · counterfactual data quality | relation accuracy (%): 76.9 -> 95.2 | Improved |
| scale generated data from 10k to 300k (ablation) | GCS fine-tuned CLIP · VL-Checklist Relation | accuracy: 74.5 -> 85.6 | Improved |
| fine-tune with GCS instead of text-perturbation methods | CLIP · Winoground | Group score: 8.0 (NegCLIP) -> 10.0 | Improved |
| fine-tune with GCS | CLIP · Elevater all-shot | accuracy: 78.96 -> 78.50 | No clear effect |
| use proposed loss instead of sigmoid loss (ablation) | CLIP fine-tuning · training time on 4 V100 | training time reduction: 16.2% less than sigmoid loss, 13.6% less than contrastive | Improved |
| fine-tune BLIP-2 with GCS-300k | BLIP-2 · ARO-R | accuracy: 41.2 -> 74.9 | Improved |
| fine-tune MiniGPT-4 with GCS-300k | MiniGPT-4 · Winoground | accuracy: 8.3 -> 15.3 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- BLIP-2 — BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.89)
- ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs (NeurIPS 2024 · similarity 0.88)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.88)
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits (NeurIPS 2025 · similarity 0.88)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.88)
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
