Research map / Vision-language and multimodal / Multimodal LLMs
Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why multimodal chain-of-thought helps LVLMs by framing "visual thoughts" as an intermediate visual cache, compares four expression types (N-LANG, S-LANG, E-IMG, G-IMG), and probes attention and information flow in LLaVA and Qwen2-VL.
Problem
The mechanisms behind textual and interleaved multimodal chain-of-thought (MCoT) gains are not well understood, and it is unclear which MCoT paradigm is better or why.
Approach
Proposes that MCoT gains come from "visual thoughts" (intermediate reasoning steps conveying image information regardless of format), defines four expression strategies for them, and analyzes their effectiveness along clarity and conciseness, plus their internal attention and information-flow behavior in LVLMs.
Key results
- Across LLaVA-1.5-7B, Qwen2-VL-7B, GPT-4o-mini and GPT-4o, all four visual thought expressions improve over w/o VT on most tasks; e.g., GPT-4o average rises from 56.83 (w/o VT) to 63.80 (E-IMG).
- Manual clarity scores correlate strongly with accuracy (Spearman and Pearson both above 0.8, P < 0.01), while fidelity scores show no significant correlation (below 0.15).
- Attention to visual thoughts exceeds attention to the original image in most layers for LLaVA and Qwen models, and attention to the raw image without visual thoughts drops sharply after layer 12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual thoughts instead of removing the visual-thought cache (ablation) | MCoT · CoMT and IsoBench | accuracy | Helped |
| Use image-form rather than text-form visual thoughts (ablation) | MCoT · CoMT-Selection | accuracy: 47.83% | Helped |
| Use visual thoughts rather than captions alone (ablation) | MCoT · Complex scenes | accuracy: over 7% | Helped |
| Choose among N-LANG, S-LANG, E-IMG, and G-IMG rather than w/o VT (ablation) | LVLMs · MMVP, V*Bench, M3CoT, and CoMT | accuracy | Helped |
| Use image-form instead of text-form visual thoughts (ablation) | MCoT · Scenarios of varying complexity | accuracy | Helped |
| Use visual thoughts for complex visual operations rather than perception tasks | LVLMs · CoMT | accuracy | Helped |
| Use T-MCoT rather than I-MCoT | LVLMs · MMVP and V*Bench-position | accuracy | Helped |
| Use I-MCoT rather than T-MCoT | LVLMs · V*Bench-attributes, M3CoT, and CoMT | accuracy | Helped |
| Increase external-tool noise in edited or generated visual thoughts (ablation) | E-IMG and G-IMG · MCoT reasoning | accuracy | Hurt |
| Intercept query-to-visual-thought information flow before layer 19 (ablation) | LLaVA · Answer prediction | accuracy | Hurt |
| Disrupt information flow from the image (ablation) | LLaVA · Answer prediction | accuracy | No clear effect |
Surprising findings
- Omitting visual thoughts can make performance worse than reasoning directly from the query.
- Visual thoughts function as a distilled cache rather than a faithful replica: fidelity scores do not correlate with accuracy.
Limitations
- Excludes multi-round visual thought interactions, which limits generality.
- Uses DALL-E 3 for G-IMG because any-to-any LVLMs struggled to generate interleaved outputs.
- Code and data are not released; some metrics (clarity, conciseness, fidelity) are manually scored, and error bars are not reported for all tasks.
Experimental setup
Datasets: MMVP V*Bench M3CoT CoMT IsoBench
Benchmarks: MMVP V*Bench M3CoT CoMT IsoBench
Metrics: accuracy Spearman correlation Pearson correlation attention score token count
Baselines: w/o VT (direct prompting without visual thoughts) N-LANG S-LANG E-IMG G-IMG
Builds on: chain-of-thought multimodal chain-of-thought Visual Sketchpad
Closest papers by meaning
- On the Faithfulness of Visual Thinking: Measurement and Enhancement (NeurIPS 2026 · similarity 0.93)
- ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.93)
- Imagine While Reasoning in Space: Multimodal Visualization-of-Thought (ICML 2025 · similarity 0.92)
- Let’s Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts (AAAI 2026 · similarity 0.92)
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.91)
- Interleaved-Modal Chain-of-Thought (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.