Research map / Vision-language and multimodal / Visual question answering
Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Defeasible Visual Entailment (DVE), a benchmark built from SNLI, Flickr30k and δ-NLI, a reference-free inference-aware evaluator trained with pairwise contrastive and categorical losses, and a critique-refine reward-driven update optimization loop on GPT-4o.
Problem
Visual entailment has focused on definite relations and does not model defeasibility, i.e., how additional updates strengthen or weaken an image-hypothesis entailment; existing generation metrics do not capture changes in entailment strength.
Approach
Train an unsupervised evaluator that scores entailment strength of (update, premise, hypothesis) triplets via pairwise contrastive comparison against caption-based triplets plus a strengthener/weakener classification head, then use its score as a reward in a critique-and-refine loop that improves LVLM-generated updates.
Key results
- GPT-4o reaches 81.76% accuracy on the DVE classification task, versus 71.10% for fine-tuned CLIP and 52.07% for zero-shot LLaVA-1.5.
- The proposed evaluator correlates with human judgments at Pearson r = 0.8262 for GPT-4o outputs, while ROUGE-L, BLEU and BERTScore correlate negatively.
- Reward-driven optimization raises GPT-4o's strengthener score from 3.6413 to 4.0679 under the evaluator, and the weakener score from -2.5212 to -2.9240.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reward-driven update optimization rather than unoptimized generation | GPT-4o strengthener generation · DVE | inference strength: 3.6413 -> 4.0679 | Helped |
| Use reward-driven update optimization rather than unoptimized generation | GPT-4o weakener generation · DVE | inference strength: -2.5212 -> -2.9240 | Helped |
| Use the inference-aware evaluator instead of CLIPScore | GPT-4o update evaluation · Human-judged update quality | Pearson's r: 0.1068 -> 0.8262 | Helped |
| Use the inference-aware evaluator instead of CLIPScore | LLaVA-1.5 update evaluation · Human-judged update quality | Pearson's r: 0.2760 -> 0.7733 | Helped |
| Set optimization threshold to ±0.5 instead of ±1.0 (ablation) | GPT-4o reward-driven update optimization · DVE generation optimization, Round3 | score: 0.9163 -> 0.9384 | Helped |
| Set optimization threshold to ±1.5 instead of ±1.0 (ablation) | GPT-4o reward-driven update optimization · DVE generation optimization, Round3 | score: 0.9163 -> 0.8670 | Hurt |
| Set optimization threshold to ±2.0 instead of ±1.0 (ablation) | GPT-4o reward-driven update optimization · DVE generation optimization, Round3 | score: 0.9163 -> 0.8005 | Hurt |
Surprising findings
- Standard generation metrics (ROUGE-L, BLEU, BERTScore) show negative correlation with human judgments of update entailment strength.
- Most zero-shot LVLMs other than GPT-4o largely fail to produce effective weakeners, mostly defaulting to captioning.
Limitations
- Evaluator validated by human study on only two generator sources (GPT-4o and LLaVA-1.5) and 203 test images.
- Dataset built from SNLI premises with Flickr30k images and δ-NLI updates, limiting diversity; classification and generation rely heavily on crowd-written updates.
Experimental setup
Datasets: SNLI Flickr30k δ-NLI
Benchmarks: DVE
Metrics: accuracy ROUGE-L BLEU-4 BERTScore CLIPScore Pearson r Spearman rho Kendall tau inference-aware evaluator score Fleiss kappa
Baselines: VILT FLAVA CLIP InstructBLIP LLaVA-1.5 mPLUG-Owl GPT-4o Multimodal-GPT MiniGPT-4 ROUGE-L BLEU BERTScore
Builds on: δ-NLI SNLI Flickr30k ResNet BERT
Closest papers by meaning
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.88)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.87)
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.86)
- Compositional Entailment Learning for Hyperbolic Vision-Language Models (ICLR 2025 · similarity 0.86)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.86)
- When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.