Research map / Vision-language and multimodal / Multimodal LLMs
HalLoc: Token-level Localization of Hallucinations for Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HalLoc, a 155K-sample token-level hallucination detection dataset across VQA, instruction-following and captioning, and HalLocalizer, a lightweight VisualBERT-based detector with per-type heads for probabilistic detection.
Problem
Existing hallucination detectors for vision-language models are computationally expensive, give binary response-level or segment-level outputs, and lack token-level localization with hallucination-type labels and calibrated probabilities.
Approach
Build a large token-level hallucination dataset by injecting GPT-4-verified hallucinated answers (from GQA scene graphs and VLM-generated answers) into source texts, then train a lightweight VisualBERT-based detector with four per-type linear heads that optionally consumes VLM hidden states to output calibrated token-level hallucination probabilities.
Key results
- On HalLoc-VQA, HalLocalizer with InternVL embeddings reaches F1 0.71 (object), 0.94 (attribute), 0.71 (relationship), 0.93 (scene).
- On HalLoc-Caption, HalLocalizer without LVLM embedding reaches F1 0.68/0.64/0.71/0.76 across the four types, beating CHAIR (object F1 0.04).
- On HalLoc-Instruct, the LogProb baseline (LLaVA) gets total F1 0.47 versus 0.91 for HalLocalizer; on Caption it gets 0.17 versus 0.71.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HalLocalizer instead of token-level log probabilities | LLaVA token hallucination detection · HalLoc-VQA | Total F1: 0.95 vs 0.95 | No clear effect |
| Use HalLocalizer instead of token-level log probabilities | LLaVA token hallucination detection · HalLoc-Instruct | Total F1: 0.47 to 0.91 | Helped |
| Use HalLocalizer instead of token-level log probabilities | LLaVA token hallucination detection · HalLoc-Caption | Total F1: 0.17 to 0.71 | Helped |
| Add LVLM embeddings to VisualBERT text input (ablation) | HalLocalizer · HalLoc-VQA | F1 across hallucination types: 0.69 -> 0.71; 0.77 -> 0.94; 0.60 -> 0.71; 0.81 -> 0.93 | Mixed |
| Add LVLM embeddings to VisualBERT text input (ablation) | HalLocalizer · HalLoc-Instruct and HalLoc-Caption | F1 across hallucination types: Instruct: 0.82 -> 0.79; Caption: 0.68 -> 0.58 | Hurt |
| Replace VisualBERT with BERT and use LVLM embeddings only (ablation) | HalLocalizer · HalLoc-VQA | F1 across hallucination types: 0.69 -> 0.72; 0.77 -> 0.95; 0.60 -> 0.70; 0.81 -> 0.93 | Helped |
| Use VisualBERT alone without LVLM embeddings (ablation) | HalLocalizer · HalLoc-Caption | F1 across hallucination types: 0.58 -> 0.68; 0.41 -> 0.64; 0.54 -> 0.71; 0.65 -> 0.71 | Helped |
Surprising findings
- Adding LVLM embeddings helped on VQA but hurt on Instruct and Caption, where the VisualBERT text-only variant was better.
- Raw VLM token log probabilities give low precision (0.32 on Instruct, 0.12 on Caption), so they are poor hallucination detectors without calibration.
Limitations
- Hallucinated tokens are very sparse in Caption (about 5%) and Instruct (about 28% of samples), making positive-label calibration hard.
- Hallucinations are synthetically injected by GPT-4 with verification, so they may not reflect naturally generated VLM errors.
- Comparisons with existing detectors are limited because other methods cannot do token-level detection.
Experimental setup
Datasets: GQA Visual Genome Stanford Image Paragraphs Localized Narratives HalLoc
Benchmarks: HalLoc-VQA HalLoc-Instruct HalLoc-Caption
Metrics: precision recall F1 ECE ACE accuracy
Baselines: CHAIR Always 1 LogProb (LLaVA token log probabilities) InternVL (calibration baseline)
Builds on: VisualBERT GQA CHAIR
Closest papers by meaning
- Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.93)
- Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.92)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
- HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models (ICML 2026 · similarity 0.92)
- Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.92)
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.