Research map / Vision-language and multimodal / Multimodal LLMs

Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies "divergent thinking" reasoning steps as a unique cause of hallucinations in multimodal chain-of-thought models and proposes a training-free, entropy-based decoding intervention (visual entropy) to mitigate them.

Problem

Multimodal chain-of-thought (MCoT) models hallucinate, producing textual answers that conflict with visual evidence; prior work attributes this to visual attention decay, which may not explain MCoT-specific hallucinations.

Approach

Hallucinations concentrate in associative "divergent thinking" steps; a normalized entropy over visual-token probabilities ("visual entropy") localizes these steps, and decoding is adaptively penalized at those steps by scaling token probabilities with exp(-alpha * visual entropy).

Key results

• On Object HalBench, the method reduces PixelReasoner-7B CHAIR_S from 22.0 to 15.4 and CHAIR_I from 7.8 to 5.3, outperforming FlexAC (18.8 CHAIR_S).
• Applied to R1-Onevision-7B, CHAIR_S drops from 23.2 to 15.8 and POPE Adversarial accuracy rises from 78.5 to 81.6.
• On MathVista, PixelReasoner-7B accuracy improves from 62.6 to 64.0 while hallucinations decrease.

Surprising findings

• Divergent thinking steps show roughly 5x higher hallucination rates than normal thinking steps.
• DoLa and VCD reduce hallucinations on PixelReasoner-7B but lower MathVista accuracy, e.g., VCD drops 0.7 points.

Limitations

• Divergent-thinking labels were produced semi-automatically with GPT-5 and verified by annotators, so the labeling depends on a proprietary model.
• Hyperparameters (gamma, alpha) were tuned mainly on PixelReasoner-7B, with claimed transfer to other models.
• Evaluation covers only three MCoT models and a limited set of benchmarks.

Experimental setup

Datasets: Object HalBench POPE MMhalBench MMStar MathVista VSR

Benchmarks: Object HalBench POPE MMhalBench MMStar MathVista VSR

Metrics: CHAIR_S CHAIR_I HalRate Score accuracy F1

Baselines: DoLa VCD MemVR FlexAC PAI

Builds on: Chain-of-Thought DoLa VCD PAI MemVR

Closest papers by meaning

• Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.93)
• More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models (NeurIPS 2025 · similarity 0.93)
• MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.92)
• Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models (NeurIPS 2025 · similarity 0.92)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.92)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.91)