Research map / Vision-language and multimodal / Multimodal LLMs

Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a token-level "grounding usage" reliability score from feature-based sensitivity and counterfactual (no-video, mismatched-video) probability margins to detect hallucinations in sign language translation, evaluated on PHOENIX-2014T and CSL-Daily with gloss-based and gloss-free models.

Problem

Hallucination in sign language translation, where fluent outputs are not supported by the signed video; existing text-only uncertainty signals (confidence, entropy, perplexity) do not reveal whether outputs are grounded in visual input.

Approach

A reliability score fusing hidden-state angular sensitivity and cross-attention usage with counterfactual probability margins (clean vs. no-video vs. mismatched-video), pooled to sentence level with tail-10% pooling, and combined with text-only signals in a META variant.

Key results

• Grounding (ours) improves over text-only baselines by up to +0.20 AUC and +0.30 ACC on hallucination detection (e.g., CSL GB AUC 0.803 vs 0.647 confidence).
• META (ours) reaches the strongest detection, e.g., AUC 0.963 / AP 0.994 on CSL GF.
• On regression, text baselines slightly lead; grounding alone reaches Spearman -0.680 on CSL GF, while META reaches -0.736.

Surprising findings

• Gloss-free models are overconfident with weak visual usage, with the correlation between reliability and hallucination stronger in gloss-free models.
• Text-only regression baselines (token entropy, perplexity) slightly outperform grounding alone on CHAIR regression.

Limitations

• CHAIR is adapted to SLT via content-word overlap, a proxy label rather than human-verified hallucination.
• Regression outputs are non-monotonic under degradation and are optimized for ranking rather than calibration.
• Evaluated on only two datasets and two SLT systems.

Experimental setup

Datasets: PHOENIX-2014T CSL-Daily

Benchmarks: PHOENIX-2014T CSL-Daily

Metrics: AUC AP ACC@0.5 Pearson Spearman ISO CHAIR BLEU chrF

Baselines: Confidence Perplexity Token entropy SelfCheckGPT

Builds on: CHAIR SpaMo TwoStream-SLT SelfCheckGPT

Closest papers by meaning

• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.91)
• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.91)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.91)
• Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding (AAAI 2026 · similarity 0.90)
• Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.90)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.90)