Research map / Vision-language and multimodal / Multimodal LLMs
Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeHall, a training-free six-step bottom-up reasoning framework (scene-graph perception, tool verification, question conflict revision, commonsense induction and web-search verification, then answering) that reduces MLLM hallucinations at perception and cognition levels.
Problem
MLLMs hallucinate both at the perception level (objects, attributes, relations) and at the cognition level (commonsense), and are often misled by hallucinated or conflicting textual questions.
Approach
A training-free, chain-of-thought-style pipeline that builds a partial scene graph, verifies it with open-set detectors (Grounding DINO, BLIP), revises conflicting input questions, induces commonsense claims verified via Google search, and answers from the verified evidence.
Key results
- On POPE adversarial, GPT-4V + framework reaches 93.9 F1 vs 82.0 baseline (+7.3).
- On PhD misleading questions, average improves by 12.3 (LLaVA-1.5), 12.1 (Qwen-VL-Chat), 19.4 (MiniGPT-V2) and 17.7 (GPT-4V) points.
- On VQAv2-IDK, total score rises from 41.97 to 55.99 for GPT-4V (+14.02) and from 3.73 to 15.08 for LLaVA-1.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Dehall | LLaVA-1.5 · POPE Adversarial | F1: 82.6 -> 91.5 | Helped |
| Apply Dehall | Qwen-VL-Chat · POPE Adversarial | F1: 82.3 -> 92.2 | Helped |
| Apply Dehall | GPT-4V · POPE Adversarial | F1: 82.0 -> 93.9 | Helped |
| Apply Dehall | LLaVA-1.5 · VQAv2-IDK | Total: 3.73 -> 15.08 | Helped |
| Apply Dehall | GPT-4V · AMBER | AMBER Score: 91.4 -> 94.5 | Helped |
| Add target identification/visual perception and visual perception verification (ablation) | Dehall with LLaVA-1.5 · PhD misleading OR | accuracy: 26.6 -> 35.4 | Helped |
| Add visual perception verification (ablation) | Dehall with LLaVA-1.5 · PhD misleading AR | accuracy: 17.3 -> 24.3 | Helped |
| Add question answering validation (ablation) | Dehall with LLaVA-1.5 · PhD misleading OR | accuracy: 24.1 -> 35.4 | Helped |
| Add commonsense verification (ablation) | Dehall with LLaVA-1.5 · PhD misleading OR | accuracy: 31.1 -> 35.4 | Helped |
| Add commonsense induction and commonsense verification (ablation) | Dehall with LLaVA-1.5 · PhD misleading OR | accuracy: 30.9 -> 35.4 | Helped |
| Use the full scene graph, including relations, instead of object+attribute only (ablation) | Dehall with LLaVA-1.5 · POPE | F1: 86.1 -> 91.5 | Helped |
| Use the full scene graph, including attributes, instead of object+relation only (ablation) | Dehall with LLaVA-1.5 · POPE | F1: 89.0 -> 91.5 | Helped |
Surprising findings
- Up to 47.8% of evaluated benchmark input questions contain hallucinated or conflicting content, and input hallucinations substantially degrade response accuracy.
- Relationship hallucinations occur more often than object and attribute hallucinations in MLLM scene graphs.
Limitations
- Multi-step reasoning adds computational complexity and may hurt quick-response scenarios.
- Performance is bounded by the base MLLM's visual understanding, especially for ambiguous or abstract images.
- Existing hallucination metrics may not fully capture improvements.
Experimental setup
Datasets: Visual Genome AMBER PhD POPE VQAv2-IDK WHOOPS! Hal-Eval
Benchmarks: POPE PhD AMBER VQAv2-IDK WHOOPS! Hal-Eval
Metrics: accuracy precision recall F1 IDK score AMBER score CHAIR BLIP score
Baselines: Woodpecker LURE LogicCheckGPT DVP MMCoT DDCoT CCoT HiKER-SGG
Builds on: chain-of-thought Woodpecker CCoT
Closest papers by meaning
- MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.94)
- Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs (ICLR 2025 · similarity 0.93)
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models (NeurIPS 2025 · similarity 0.93)
- Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models (CVPR 2026 · similarity 0.93)
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.93)
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.