Research map / Vision-language and multimodal / Multimodal LLMs

Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Instruction Lens Score (InsLen), a training-free plug-and-play object hallucination detector that combines a calibrated local visual score with an instruction-embedding-based context consistency score, outperforming prior detectors on MSCOCO, Objects365, POPE and CLEVR.

Problem

Detecting object hallucinations in MLLM outputs without costly auxiliary models, since vision-based internal scores can be spuriously overconfident due to misleading visual embeddings.

Approach

Instruction-token embeddings, read out via Logit Lens, encode grounded object cues and suppress misleading visual evidence; InsLen uses them to calibrate a patch-level local similarity score (Cafe multiplicative factor) and to build a global context consistency score between the object token embedding and averaged top-m instruction embeddings.</'core_idea>

Key results

• On MSCOCO with Qwen3-VL-8B, InsLen reaches 81.02% AUROC vs 73.32% for GLSIM (best baseline).
• On POPE with LLaVA-1.5-7B, InsLen achieves 83.94% AUROC, up to 13.81 points above the strongest baseline.
• On CLEVR with Qwen3-VL-8B, InsLen reaches 77.72% AUROC vs 73.01% for SVAR.

Surprising findings

• Instruction-embedding confidence outperforms image-embedding confidence by at least 8% AUROC on LLaVA-1.5, suggesting instruction tokens filter misleading visual evidence.
• For InstructBLIP, where a Q-Former fuses vision before the LLM, image-embedding confidence outperforms instruction-embedding confidence (80% vs 72% AUROC), a limitation of the approach.

Limitations

• Degrades on InstructBLIP because Q-Former fuses visual and instruction information before the LLM.
• Logit Lens reveals visual information often encoded as synonyms rather than the exact generated object words, limiting matching.
• Slightly higher latency than some baselines, though small relative to answer generation.

Experimental setup

Datasets: MSCOCO Objects365 POPE CLEVR

Benchmarks: MSCOCO Objects365 POPE CLEVR

Metrics: AUROC AUPR

Baselines: NLL Entropy Internal Confidence SVAR Contextual Lens EAZY GLSIM

Builds on: Logit Lens Local Similarity Score SVAR Internal Confidence CHAIR

Closest papers by meaning

• Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.92)
• Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding (AAAI 2026 · similarity 0.91)
• Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.91)
• Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.91)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.91)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.90)