Research map / Vision-language and multimodal / Multimodal LLMs
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the 438k-sample Visual CoT dataset with intermediate bounding boxes (about 98k with reasoning steps), a two-turn VisCoT pipeline that crops the predicted region and re-encodes it, and a Visual CoT benchmark for MLLMs.
Problem
Existing MLLMs process whole images at fixed resolution as black boxes, which hurts performance on high-resolution inputs and questions about small regions, and offers little interpretability; there is no visual chain-of-thought supervision in standard VQA datasets.
Approach
Annotate VQA pairs with key-region bounding boxes (and GPT-4-generated reasoning steps for GQA) and train an MLLM to first predict a CoT bbox, then crop that region from the original image and integrate both the global and local visual tokens to answer.
Key results
- VisCoT-7B at 336 res reaches average 0.580 on the Visual CoT benchmark vs 0.454 for LLaVA-1.5-7B; SROIE improves from 0.136 to 0.470 for the 336 model.
- Ground-truth CoT boxes raise average benchmark score to 0.752, showing box prediction accuracy is the main bottleneck.
- VisCoT-7B at 336 res achieves 87.46 / 92.05 / 81.18 on RefCOCO+ val/testA/testB, outperforming listed generalist models including Qwen-VL-7B and SPHINX-13B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual CoT (bbox-crop second pass) versus direct answering (ablation) | VisCoT-7B (224^2) · Visual CoT benchmark, average | average score: 0.443 -> 0.550 | Helped |
| use ground-truth bbox instead of predicted bbox (ablation) | VisCoT-7B · Visual CoT benchmark, average | average score: 0.550 -> 0.752 (upper bound) | Helped |
| replace predicted bbox with random bbox (ablation) | VisCoT-7B · Visual CoT benchmark, average | average score: 0.550 -> 0.453 | Hurt |
| replace predicted bbox with center bbox (ablation) | VisCoT-7B · Visual CoT benchmark, average | average score: 0.550 -> 0.490 | Hurt |
| add expanded and centered cropping in visual sampler (ablation) | VisCoT-7B visual sampler · Visual CoT benchmark, average | average score: 0.496 -> 0.550 | Helped |
| add expanded cropping only (ablation) | VisCoT-7B visual sampler · Visual CoT benchmark, average | average score: 0.496 -> 0.506 | Helped |
| add centered cropping only (ablation) | VisCoT-7B visual sampler · Visual CoT benchmark, average | average score: 0.496 -> 0.518 | Helped |
| increase input resolution from 224 to 336 with visual CoT | VisCoT-7B · Visual CoT benchmark, average | average score: 0.550 -> 0.580 | Helped |
| apply visual CoT (vs. direct answer) on high-resolution doc/text tasks (ablation) | VisCoT-7B · SROIE and DocVQA/TextVQA (Visual CoT benchmark) | score: SROIE 0.044 w/o CoT vs 0.341 with CoT (224) | Helped |
| add visual CoT inference on VQA benchmarks (ablation) | VisCoT-7B · DocVQA, TextVQA, ChartQA | accuracy: DocVQA 29.4 -> 49.3 (336); TextVQA 60.2 -> 66.9; ChartQA 17.5 -> 22.8 | Helped |
| add visual CoT on Visual CoT benchmark, bbox-relative-size analysis | VisCoT-7B · Visual CoT benchmark, per-dataset | accuracy gain vs. no-CoT | Helped |
| direct answering (no CoT) on standard MLLM benchmarks versus LLaVA-1.5 | VisCoT-7B · MMBench, POPE, SQA, GQA | accuracy: comparable to LLaVA-1.5 (e.g., MMBench 67.9 vs 64.3 for 7B, POPE 86.5 vs 85.9) | No clear effect |
Surprising findings
- Random bbox selection performs about the same as no-CoT inference, while ground-truth boxes give large gains.
- Visual CoT yields larger improvements when the queried region is relatively small.
Limitations
- The model may fail to identify the relevant region in images with extensive information or complex questions, leading to wrong answers.
- Dataset annotations depend on GPT-4 outputs, which may contain inaccuracies and inherited biases.
- The benchmark relies on ChatGPT-based scoring, and no error bars are reported.
Experimental setup
Datasets: TextVQA TextCaps DocVQA DUDE SROIE Birds-200-2011 Flickr30k Visual7W InfographicsVQA VSR GQA Open Images
Benchmarks: Visual CoT benchmark SQA TextVQA MME POPE MMBench MMBench-Chinese GQA RefCOCO RefCOCO+ RefCOCOg DocVQA
Metrics: ChatGPT-judged score (0-1) Top-1 Accuracy@0.5 accuracy
Baselines: LLaVA-1.5-7B LLaVA-1.5-13B SPHINX-13B Qwen-VL Shikra InstructBLIP BLIP-2 IDEFICS
Builds on: LLaVA-1.5 Shikra CLIP Vicuna
Closest papers by meaning
- KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning (AAAI 2024 · similarity 0.91)
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.91)
- Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models (AAAI 2024 · similarity 0.91)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.90)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.