Research map / Vision-language and multimodal / Multimodal LLMs
Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces R-Bench, a nocaps-based yes/no benchmark (image-level and instance-level with box/mask prompts) that evaluates relationship hallucinations in LVLMs and analyzes their co-occurrence and long-tail causes.
Problem
LVLMs are known to hallucinate objects, but hallucinations about inter-object relationships are under-evaluated, and existing relationship benchmarks risk data leakage from COCO/Visual Genome overlap with LVLM training data.
Approach
A relationship hallucination benchmark built on the nocaps validation set (to avoid training leakage), with LLM-generated yes/no questions seeded by scene-graph triplets from COCO captions, plus instance-level questions that mark subject and object with colored boxes or SAM masks, followed by manual filtering and co-occurrence analysis.
Key results
- On image-level R-Bench, Qwen-VL reaches 79.19% accuracy while LLaVA-1.5 reaches 71.23%, but on instance-level box questions all models stay near 51-59% accuracy.
- Relationship hallucination is more severe than object hallucination: on the same nocaps images, LLaVA-1.5 gets 78.34% accuracy on POPE object questions versus 71.23% on image-level relationship questions.
- On counterfactual web images, Qwen-VL achieves the best total accuracy of 30%, and the best illusion accuracy is 25% (InstructBLIP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instance-level (box/mask) relationship questions vs image-level questions | LVLMs on R-Bench · R-Bench | Accuracy: image-level ~69-79 vs instance-level ~52-60 | Hurt |
| Relationship hallucination vs object hallucination (image-level questions) | LVLMs evaluated with POPE-style questions · nocaps validation (POPE vs R-Bench) | Accuracy: e.g., LLaVA-1.5 78.34 object vs 71.23 relationship | Hurt |
| Counterfactual and illusion relationship questions on web images | LVLMs · web-collected images (about 20) | Total accuracy: best 30.00 (Qwen-VL) | Hurt |
| Red/green colour swap for visual markers (ablation) | LVLMs on box-based instance questions · R-Bench Box | Accuracy: minor changes, e.g., LLaVA-1.5 53.15 to 53.52 | No clear effect |
| Gold and pink colours instead of red and green for markers (ablation) | LVLMs on box-based instance questions · R-Bench Box | Accuracy: Qwen-VL 58.82 to 57.97; InstructBLIP 51.95 to 52.22 (mixed); mPLUG-Owl2 53.90 to 52.97 | Hurt |
| Relationship-relationship, subject-relationship, relationship-object co-occurrence in training data | LLaVA-1.5 hallucination analysis · R-Bench | hallucination probability | Hurt |
Surprising findings
- Models answer image-level relationship questions well but fail the same relationships when instance-level visual prompts are used.
- LVLMs answer based on LLM common sense over visible content, e.g., misjudging a one-wheeled motorcycle as having two wheels.
Limitations
- Final benchmark relies on heavy manual filtering, which may still leave noise and bias.
- Counterfactual and illusion analyses use only about 20 web-collected images, so results are small-sample.
- Closed-source LVLMs such as GPT-4 were not evaluated due to budget constraints.
Experimental setup
Datasets: nocaps validation set COCO captions OpenImages
Benchmarks: R-Bench POPE
Metrics: accuracy precision recall F1 score yes ratio
Baselines: LLaVA-1.5 InstructBLIP mPLUG-Owl2 Qwen-VL
Builds on: POPE GroundingDINO SAM scene graph parser
Closest papers by meaning
- ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations (ICLR 2026 · similarity 0.92)
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.92)
- Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.92)
- THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models (CVPR 2024 · similarity 0.91)
- MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.91)
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.