Research map / Vision-language and multimodal / Multimodal LLMs
NL-Eye: Abductive NLI For Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NL-Eye, a 350-triplet synthetic image benchmark testing VLMs' visual abductive NLI (plausibility prediction and explanation), finding VLMs near or below random while humans excel.
Problem
Whether modern vision-language models can perform abductive reasoning over multiple images, i.e., infer which hypothesis image is more plausible given a premise image and explain why.
Approach
Adapts textual abductive NLI to the visual domain with premise and hypothesis images generated by text-to-image models, evaluating both plausibility prediction (triplet and pairs setups with order-consistency) and explanation quality via human and LLM-judge evaluation.
Key results
- Humans reach 85% triplet prediction accuracy and 95% explanation validity, while most VLMs fall below the 50% dumb-pixel baseline; only GPT-4o reaches 60% on combined-image triplets.
- Text-only reasoning with gold descriptions gives around 80% accuracy for GPT-4o and Claude models, showing the bottleneck is visual interpretation.
- VLM predictions are highly sensitive to hypothesis order, e.g., GPT-4o scores 97% when the correct hypothesis is first but 16% when second in separate-image triplets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use combined-image rather than separate-image input | GPT-4o on NL-Eye · NL-Eye vision, triplet | consistency accuracy: 16% -> 60% | Helped |
| Use separate-image rather than combined-image input | Gemini-1.5-Pro, GPT-4-Vision, and Claude-Sonnet-3.5 · NL-Eye vision, triplet | consistency accuracy: 48.6% vs. 37.3% | Helped |
| Use gold textual descriptions instead of model-generated image captions | GPT-4o predictor on NL-Eye · NL-Eye triplet | consistency accuracy: 80% vs. 29%-40% | Helped |
| Replace visual inputs with gold textual descriptions | GPT-4o on NL-Eye · NL-Eye triplet | consistency accuracy: 60% -> 80% | Helped |
| Use combined-image rather than separate-image input | GPT-4o on NL-Eye · NL-Eye vision, pairs | order-faithful accuracy: 50% -> 45% | Hurt |
| Use separate-image rather than combined-image input | VLMs on NL-Eye · NL-Eye vision, pairs | order-faithful accuracy: GPT-4-Vision: 34% -> 40%; Gemini-1.5-Pro: 39% -> 42%; GPT-4o: 45% -> 50% | Mixed |
| Use text-only reasoning instead of vision-based reasoning | VLMs on NL-Eye · NL-Eye triplet | consistency accuracy: around 80% vs. vision-based scores up to 60% | Helped |
| Compare two hypotheses rather than score a single hypothesis | VLMs on NL-Eye · NL-Eye vision | prediction accuracy | Mixed |
| Present the correct hypothesis first rather than second | Gemini-1.5-Pro, combined-image input · NL-Eye vision, triplet | order-specific accuracy: 82% -> 46% | Hurt |
| Present the correct hypothesis second rather than first | GPT-4o, separate-image input · NL-Eye vision, triplet | order-specific accuracy: 97% -> 16% | Hurt |
| Generate explanations with a VLM rather than a human | NL-Eye explanation task · NL-Eye vision, correct predictions | human explanation validity: 95% vs. at best 50% | Hurt |
| Reason about parallel rather than forward or backward temporal relations | VLMs on NL-Eye · NL-Eye temporal categories | prediction accuracy | Helped |
Surprising findings
- GPT-4o predicts the first hypothesis as plausible almost always, so its accuracy reflects order bias rather than reasoning.
- Even when VLMs predict correctly, over half of their explanations are judged invalid by human annotators.
Limitations
- Small benchmark of 350 triplets, curated with heavy human involvement.
- Images are synthetic (Midjourney/DALL-E 3) rather than natural photographs.
- Automatic explanation evaluation relies on a GPT-4o judge comparing to reference explanations.
Experimental setup
Datasets: NL-Eye
Benchmarks: NL-Eye
Metrics: consistency accuracy order-faithfulness accuracy human validity rate for explanations LLM-judge explanation alignment
Baselines: Gemini-1.5-Pro GPT-4 Vision GPT-4o Claude-3.5-Sonnet Claude-3-Opus LLaVA-1.6 Fuyu-8b BART-L-MNLI DeBERTa-v3-nli random baseline
Builds on: abductive NLI visual entailment
Closest papers by meaning
- AbductiveMLLM: Boosting Visual Abductive Reasoning Within MLLMs (AAAI 2026 · similarity 0.91)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.89)
- VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning? (ICLR 2026 · similarity 0.89)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.88)
- ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.88)
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.