Research map / Vision-language and multimodal / Multimodal LLMs
DOUBT: Decoupled Object-level Understanding and Bridging via vMF-based Trustworthiness for Hallucination Detection in MLLMs
Kaiqi Chen, Yang Qin, Changhao He, Xi Peng, Peng Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOUBT, a hallucination detection method for MLLMs that decouples object-level visual understanding from language and uses vMF-based trustworthiness scores.
Problem
Hallucinations in multimodal LLMs (assertions contradicting the image or facts) undermine reliability; existing detectors feed image and text jointly and measure output consistency, but the visual module often lags behind the language module in understanding.
Approach
Decouples object-level understanding from bridging with the language side and estimates trustworthiness using von Mises-Fisher (vMF)-based modeling, rather than jointly feeding image and text and measuring output consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add object-level understanding and bridging (OUB) (ablation) | DOUBT · MMMU (avg over 10 MLLMs) | detection accuracy (%): 59.77 -> 60.22 | Improved |
| add object-level understanding and bridging (OUB) (ablation) | DOUBT · ScienceQA (avg over 10 MLLMs) | detection accuracy (%): 75.93 -> 76.07 | No clear effect |
| add object-level understanding and bridging (OUB) (ablation) | DOUBT · MMMU (Qwen2VL-2B) | detection accuracy (%): 60.85 vs 61.94 | Hurt |
| add vMF-based trustworthiness in place of semantic entropy (ablation) | DOUBT · MMMU (avg over 10 MLLMs) | detection accuracy (%): 60.01 -> 60.22 | Improved |
| add vMF-based trustworthiness in place of semantic entropy (ablation) | DOUBT · ScienceQA (avg over 10 MLLMs) | detection accuracy (%): 67.10 -> 76.07 | Improved |
| add both OUB and vMF trustworthiness (ablation) | DOUBT · MMMU (avg over 10 MLLMs) | detection accuracy (%): 56.70 -> 60.22 | Improved |
| add both OUB and vMF trustworthiness (ablation) | DOUBT · ScienceQA (avg over 10 MLLMs) | detection accuracy (%): 62.03 -> 76.07 | Improved |
| increase sampling number K beyond 10 (ablation) | DOUBT · MMVet (Qwen2VL-2B) | detection accuracy (%) | No clear effect |
| increase top-p (ablation) | DOUBT · MMVet (Qwen2VL-2B) | detection accuracy (%) | Improved |
| two-stage OUB design versus VL-Uncertainty | DOUBT · LLaVABench (Qwen2VL-2B) | time per sample (s): 9.82 -> 34.01 | Hurt |
Closest papers by meaning
- Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding (AAAI 2026 · similarity 0.93)
- Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.92)
- Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.92)
- Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention (AAAI 2026 · similarity 0.92)
- HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models (ICML 2026 · similarity 0.92)
- Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
