Research map / Vision-language and multimodal / Multimodal LLMs

Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VBackChecker, a reference-free MLLM hallucination detector that grounds each response sentence back to the image with a LISA-style segmentation model (emitting [SEG] or [REJ]), trained on a new R-Instruct data pipeline, and evaluated on a new R²-HalBench benchmark.

Problem

Detecting hallucinations in rich-context MLLM responses without references or external expert models, with interpretability in both visual and language modalities.

Approach

Verify response sentences by "looking back" at the image: a pixel-level grounding LLM outputs a [SEG] token and mask if the described element can be grounded, or a [REJ] token with a natural-language explanation if it cannot; trained with up-weighted [SEG]/[REJ] cross-entropy and a rich-context instruction dataset with hard negatives.

Key results

• On R²-HalBench, the 7B model reaches 62.5% accuracy, outperforming open-source baselines and approaching GPT-4o (63.9% acc, 68.3% with text explanation).
• On gRefCOCO validation, reaches 80.2 IoU and 81.3 N-acc, improving over prior grounding methods such as GSVA-7B (66.5 IoU).
• Removing the R-Instruct data drops R²-HalBench negative accuracy from 64.1 to 2.3, showing the data is critical for rejection.

Surprising findings

• Removing the grounding loss raises negative accuracy on R²-HalBench (81.3 vs 64.1) but drops positive accuracy sharply (33.4 vs 60.9), indicating a trade-off in rejection behavior.
• Open-source MLLMs such as LLaVA-OneVision and Qwen2-VL, used directly as detectors, show strongly skewed accuracy (near-0 positive accuracy for LLaVA-OneVision in the non-explanation setting), suggesting bias in binary yes/no judgments.

Limitations

• Benchmark positive/negative distribution and accuracy remain modest (about 62.5% overall), leaving substantial gap to GPT-4o in some settings.
• Relies on a fixed 7B LLaVA/SAM backbone and synthetic hallucination injection via Qwen2-VL-72B and Claude-3.5 for training negatives.

Experimental setup

Datasets: SA-1B gRefCOCO R-Instruct

Benchmarks: R²-HalBench POPE gRefCOCO R-Instruct-A Val

Metrics: accuracy negative accuracy positive accuracy IoU T-acc

Baselines: GPT-4o FaithScore GSVA-7B GSVA-13B LLaVA-OneVision Qwen2-VL ReLA LISA-13B

Builds on: LISA GSVA SAM FaithScore

Closest papers by meaning

• Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.93)
• Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)