Research map / Vision-language and multimodal / Multimodal LLMs

When to Think and When to Look: Uncertainty-Guided Lookback

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically compares instruct vs thinking modes of InternVL3.5 and Qwen3-VL LVLMs on MMMU and proposes a training-free uncertainty-guided lookback decoding strategy that inserts image-referring phrases during thinking, improving accuracy and reducing tokens.

Problem

Test-time thinking (explicit reasoning chains) is assumed to help LVLM visual reasoning, but there is no systematic analysis of when it helps, how breadth (sampling) and depth (thinking) trade off, and how to control thinking to stay grounded in the image.

Approach

Uses a token-level visual sensitivity probe (perplexity differences between real, noise, and no image) to mine image-referring "lookback" phrases and trigger them during streaming decoding when the chain is visually uncertain, optionally with parallel branch selection by visual helpfulness score; no training required.

Key results

• On MMMU val, the 4B Qwen3-VL lookback variant raises Pass@1 from 59.3% to 61.6% using about 57% of the original tokens, and the sampling variant reaches about 62.0%.
• On MathVista, the 4B Qwen3-VL lookback+sampling variant improves from 79.5 to 85.0 (+5.5).
• The 32B Qwen3-VL lookback variant reaches 81.7 on overall MMMU Pass@1 versus 75.3 for the original thinking model (+6.4).

Surprising findings

• Long thinking chains often produce long-wrong trajectories that ignore the image and underperform the same models in standard instruct mode.
• Periodic (fixed-interval) lookback insertion is worse than uncertainty-triggered insertion, with periodic lookback at times falling below the original thinking baseline.

Limitations

• Requires token-level log-probabilities for probe construction and trigger mining, so it does not apply to closed-source models without log-prob access.
• Analysis limited to two open-source model families (InternVL3.5 and Qwen3-VL) and mainly MMMU validation.

Experimental setup

Datasets: MMMU validation set (used as pseudo ground truth for mining phrases)

Benchmarks: MMMU MMBench MMStar MathVista MathVision MathVerse

Metrics: Pass@1 Pass@k token usage percentage accuracy

Baselines: DEER DeepConf REFRAIN Original thinking (Qwen3-VL) InternVL3.5 instruct

Builds on: DEER DeepConf REFRAIN

Closest papers by meaning

• CoFFT: Chain of Foresight-Focus Thought for Visual Language Models (NeurIPS 2025 · similarity 0.92)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
• Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.92)
• What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning (AAAI 2026 · similarity 0.91)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.91)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)