Research map / Vision-language and multimodal / Multimodal LLMs

Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes patch-level token grounding features (Attention Dispersion Score and Cross-modal Grounding Consistency) across LVLM layers, fed to lightweight classifiers to detect token-level object hallucinations.

Problem

Token-level object hallucination detection in LVLMs; existing detectors use global whole-image attention or similarity statistics, which miss hallucinated tokens that have weak but scattered correlations across many patches.

Approach

Measures per-layer spatial compactness of text-to-patch attention (ADS, via top-k foreground 8-connected components and background entropy) and top-k cosine similarity between token hidden states and patch embeddings (CGC), then trains XGB/RF/MLP classifiers on the concatenated per-layer features.

Key results

• On MS COCO captioning, Ours (XGB) reaches F1 0.82 on LLaVA-1.5-7B, 0.84 on Qwen2.5-VL-7B and 0.90 on InternVL2.5-8B, outperforming baselines such as SVAR and DHCP.
• On POPE with LLaVA-1.5-7B, the method attains F1 0.41 and AUC 0.75, versus 0.38 F1 for the best baseline ProjectAway.
• Combining ADS and CGC improves AUC over either feature alone (e.g., LLaVA-1.5 0.92 vs 0.86/0.84), and adding them to SVAR raises its AUC up to 89.41 on LLaVA-1.5.

Surprising findings

• Hallucinated tokens are reported to show lower attention to every individual patch while also scattering weak correlations, and the paper argues the visual encoder preserves structure so hallucinations stem from language priors rather than the encoder.
• Global attention-based SVAR gets AUC 0.37 on POPE, below chance, indicating global statistics fail on the POPE yes/no format.

Limitations

• POPE evaluation is heavily imbalanced (about 6% hallucinated) and yields low F1 values (0.41 at best).
• Labels for MS COCO are produced by GPT-4o, and the hallucination labeling depends on GPT-4o judgments without human verification reported.
• Hyperparameters such as top-x% and tau_ADS are chosen by sensitivity tables on one model, with limited generalization analysis.

Experimental setup

Datasets: MS COCO 2014 POPE

Benchmarks: MS COCO 2014 captioning (token-level hallucination labels) POPE

Metrics: precision recall F1 AUC

Baselines: MetaToken HalLoc SVAR DHCP ProjectAway

Builds on: SVAR MetaToken HalLoc DHCP

Closest papers by meaning

• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.94)
• Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding (AAAI 2026 · similarity 0.93)
• Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.93)
• Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing (NeurIPS 2025 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• HalLoc: Token-level Localization of Hallucinations for Vision Language Models (CVPR 2025 · similarity 0.93)