Research map / Vision-language and multimodal / Multimodal LLMs

Grounded Chain-of-Thought for Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Grounded Chain-of-Thought (GCoT) and the MM-GCoT dataset (23,028 train / 994 test examples) to measure and reduce visual hallucination in MLLMs via step-wise grounding, evaluating 12 MLLMs and an SFT-trained LLaVA GCoT.

Problem

MLLMs often answer correctly without relying on the relevant visual evidence (language bias) or answer incorrectly while grounding wrong regions, so answers and grounding are inconsistent, undermining trustworthy use in settings like embodied AI.

Approach

Train MLLMs to produce step-by-step reasoning with bounding-box grounding before the final answer, and evaluate with answer accuracy, grounding accuracy (Acc@0.5), and answer-grounding consistency metrics that expose hallucination.

Key results

• Most MLLMs show low answer-grounding consistency; LLaVA-OneVision-72B reaches 75.7% average accuracy but only 15.8% consistency under answer-first prompting.
• Qwen2.5-VL-7B averages 56.8% consistency (answer-first) vs 39.0% for Qwen2.5-VL-72B-Instruct accuracy-consistency comparison, showing larger models are not less hallucinating.
• LLaVA-7B GCoT improves average consistency from 10.1% to 58.1% and accuracy from 70.0% to 74.5%.

Surprising findings

• Larger models (e.g., Qwen2.5-VL-72B) show worse answer-grounding consistency than their 7B counterparts.
• Models can answer correctly while grounding irrelevant regions, indicating language bias.

Limitations

• Test samples validated manually but training data verified only by LLM, possibly noisy.
• Evaluation restricted to three question types (Attribute, Judgement, Object) and answer-text matching.
• Compute and training hyperparameters for LLaVA GCoT not detailed.

Experimental setup

Datasets: Visual Genome MM-GCoT trainval

Benchmarks: MM-GCoT

Metrics: answer accuracy grounding accuracy (Acc@0.5) answer-grounding consistency

Baselines: LLaVA-7B LLaVA-13B LLaVA-OneVision-0.5B LLaVA-OneVision-7B LLaVA-OneVision-72B InternVL2.5-4B InternVL2.5-8B InternVL2.5-38B InternVL2.5-78B Qwen2.5-VL-3B-Instruct Qwen2.5-VL-7B-Instruct Qwen2.5-VL-72B-Instruct

Builds on: LLaVA Visual Genome chain-of-thought prompting

Closest papers by meaning

• See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs (CVPR 2026 · similarity 0.92)
• PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs (ICLR 2026 · similarity 0.92)
• Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.91)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.91)
• More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models (NeurIPS 2025 · similarity 0.91)
• Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.90)