Research map / Vision-language and multimodal / Multimodal LLMs
Evaluating and Mitigating Relationship Hallucinations in Large Vision-Language Models
Mingrui Wu, Jiale Li, Jiayi Ji, Fei Hao, Xiaoshuai Sun, Rongrong Ji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| ask relationship questions about marked instances (box) instead of whole image | LLaVA-1.5 · R-Bench | accuracy: 71.23 -> 53.15 | Hurt |
| ask relationship questions about marked instances (box) instead of whole image | Qwen-VL · R-Bench | accuracy: 79.19 -> 58.82 | Hurt |
| ask relationship questions about marked instances (mask) instead of whole image | InstructBLIP · R-Bench | accuracy: 69.31 -> 55.63 | Hurt |
| probe relationships instead of objects | LLaVA-1.5 · nocaps (image-level) | accuracy: 78.34 -> 71.23 | Hurt |
| probe relationships instead of objects | Qwen-VL · nocaps (image-level) | accuracy: 86.57 -> 79.19 | Hurt |
| use masks instead of boxes to mark instances | Qwen-VL · R-Bench instance-level | accuracy: 58.82 -> 59.84 | Improved |
| swap red and green marker colors (ablation) | LLaVA-1.5 · R-Bench box | accuracy: 53.15 -> 53.52 | No clear effect |
| swap red and green marker colors (ablation) | Qwen-VL · R-Bench box | accuracy: 58.82 -> 59.48 | No clear effect |
| replace red/green with gold/pink (ablation) | Qwen-VL · R-Bench box | accuracy: 58.82 -> 57.97 | Hurt |
| replace red/green with gold/pink (ablation) | mPLUG-Owl2 · R-Bench box | accuracy: 53.90 -> 52.97 | Hurt |
| frequent co-occurrence of relationships with a given relationship | LLaVA-1.5 · R-Bench | hallucination probability | Hurt |
Closest papers by meaning
- Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models (ICML 2024 · similarity 0.99)
- Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.92)
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.92)
- ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations (ICLR 2026 · similarity 0.92)
- THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models (CVPR 2024 · similarity 0.92)
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
