Research map / Vision-language and multimodal / Multimodal LLMs

On the Faithfulness of Visual Thinking: Measurement and Enhancement

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RFT-trained thinking-with-images models ignore their visual evidence in multimodal CoT and proposes an LVLM-judged faithfulness metric plus SCCM reward (sufficiency times minimality) that improves visual faithfulness on V* and HR-Bench.

Problem

Multimodal chain-of-thought traces from RFT-trained LVLMs that use zoom-in tools often contain inaccurate visual information and their predictions are insensitive to visual corruption, indicating unfaithful visual reasoning.

Approach

Intervention-based causal tests show visual evidence is largely ignored; a reward that requires cropped visual evidence to be independently sufficient for the correct answer (checked by a judge LVLM) and minimal in tokens (group-relative) is added to RFT, annotation-free and plug-and-play.

Key results

• Intervening on visual crops (replacing with noise) changes V* Bench average accuracy by only about -0.5 for DeepEyes and +0.55 for Pixel-Reasoner, while textual mistake injection drops it by 9 to 18 points.
• SCCM-trained model reaches 61.26 visual reliability and 75.92 visual sufficiency on V* Bench average, versus 26.23 and 40.98 for Pixel-Reasoner.
• SCCM achieves 91.10 accuracy on V* Bench, above DeepEyes (89.00) and Pixel-Reasoner (85.86).

Surprising findings

• Curiosity-driven reward (rewarding presence of visual cues) leads to a collapse in visual sufficiency, with V* sufficiency near 8 percent.
• Removing the minimality term makes the model crop the entire image and call tools repeatedly, which hacks the sufficiency reward and destabilizes training.

Limitations

• Visual sufficiency and reliability are judged by GPT-4o or Qwen2.5-VL-72B, so the metric depends on judge reliability and judge noise.
• Intervention on textual components uses GPT-4o-injected mistakes, and visual intervention uses random noise, which may not capture all causal pathways.
• SCCM only partially mitigates visual unfaithfulness; visual intervention effects remain non-significant for the proposed model on most splits.

Experimental setup

Datasets: V* Bench HR-Bench DeepEyes RL dataset Pixel-Reasoner SFT dataset Pixel-Reasoner RL dataset

Benchmarks: V* Bench HR-Bench 4K HR-Bench 8K

Metrics: accuracy McNemar test p-value visual reliability visual sufficiency cropped region size tool call count

Baselines: DeepEyes Pixel-Reasoner SEAL Curiosity reward

Builds on: GRPO Pixel-Reasoner DeepEyes Sufficient-Component Cause Model

Closest papers by meaning

• Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.93)
• On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs (ICML 2026 · similarity 0.92)
• To Think or Not To Think: A Study of Thinking in Rule-Based Visual Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.91)
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.90)
• When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.90)
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.90)