Research map / Vision-language and multimodal / Multimodal LLMs
Natural Language Inference Improves Compositionality in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CECE, which uses an LLM (Llama3.1 70B) to generate NLI entailments and contradictions of a caption and scores them with VLMs (LLaVA-1.5/1.6) via yes/no likelihoods, balancing with the original caption score, without fine-tuning.
Problem
Compositional reasoning in VLMs is weak on binary image-text matching benchmarks (Winoground, EqBen); existing LLM-based sentence decomposition methods stay lexically close to the caption and inject LLM biases/wrong assumptions.
Approach
Use natural language inference to have an LLM generate both entailments (scored for "yes") and contradictions (scored for "no") of a caption premise, then aggregate their VLM likelihoods with a weighted balance against the original caption score, yielding lexically diverse yet meaning-preserving evaluation without fine-tuning.
Key results
- CECE with LLaVA-1.5 and LLaVA-1.6 soft-ensemble reaches 47.5 Winoground group score and 47.9 EqBen group score, without fine-tuning.
- On Winoground, CECE* gains +19.2% group score over the best prior work (finetuned DSG), per the abstract.
- On text-to-image human-alignment benchmarks, CECE* achieves 88.2 on DrawBench and 76.4 on EditBench AUROC, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contradictions to entailments (ablation) | Cece on Winoground · Winoground | group score: 36.0 -> 39.0 | Helped |
| Add contradictions to entailments (ablation) | Cece on EqBen · EqBen | group score: 33.6 -> 38.6 | Helped |
| Add the original caption to entailments and contradictions (ablation) | Cece on Winoground · Winoground | group score: 39.0 -> 42.8 | Helped |
| Add the original caption to entailments and contradictions (ablation) | Cece on EqBen · EqBen | group score: 38.6 -> 47.1 | Helped |
| Balance LLaVA-1.5 entailment/contradiction scores with LLaVA-1.6 caption scores rather than InstructBLIP + LLaVA-1.5 (ablation) | Cece soft ensemble on Winoground · Winoground | group score: 41.0 -> 47.5 | Helped |
| Balance LLaVA-1.5 entailment/contradiction scores with LLaVA-1.6 caption scores rather than InstructBLIP + LLaVA-1.5 (ablation) | Cece soft ensemble on EqBen · EqBen | group score: 40.7 -> 47.9 | Helped |
| Use LLaVA-1.6 rather than InstructBLIP for the caption scores while retaining LLaVA-1.5 for expansions (ablation) | Cece soft ensemble on Winoground · Winoground | group score: 42.5 -> 47.5 | Helped |
| Use LLaVA-1.6 rather than InstructBLIP for the caption scores while retaining LLaVA-1.5 for expansions (ablation) | Cece soft ensemble on EqBen · EqBen | group score: 40.0 -> 47.9 | Helped |
| Expand captions with entailments and contradictions instead of scoring the original caption alone | Cece on StanfordT23D · StanfordT23D | Pearson: 55.8 -> 57.4 | Helped |
| Expand captions with entailments and contradictions instead of scoring the original caption alone | Cece on StanfordT23D · StanfordT23D | pairwise accuracy: 64.9 -> 65.3 | Hurt |
Surprising findings
- Contradictions alone introduce some incorrect negative statements that can outweigh entailments and mislead the VLM on still images, causing failures in some cases.
Limitations
- Contradictions can introduce incorrect or overly negative statements, especially for still images, leading to wrong conclusions.
- Semantic drift from LLM-generated expansions requires balancing hyperparameters (alpha1, alpha2) tuned on the same benchmarks.
- Performance depends on VLM quality; Cece on BLIP-2 and InstructBLIP underperforms the base VQAScore models in some cases.
Experimental setup
Datasets: Winoground EqBen DrawBench EditBench COCO-T2I TIFA160 Pick-a-Pic StanfordT23D
Benchmarks: Winoground EqBen DrawBench EditBench COCO-T2I TIFA160 Pick-a-Pic StanfordT23D
Metrics: text score image score group score AUROC pairwise accuracy Pearson Kendall binary accuracy
Baselines: CLIPScore BLIP2 ITM VQAScore VIEScore GPT4V-Eval VisProg ViperGPT VPEval VQ2 DSG CCoT
Builds on: VQAScore DSG VQ2 CCoT
Closest papers by meaning
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.91)
- ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs (NeurIPS 2024 · similarity 0.91)
- Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.91)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.90)
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.