Research map / Vision-language and multimodal / Multimodal LLMs

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TVI-CoT, which trains Qwen3-VL-8B with learnable Think/Look/Answer control tokens so the model can interleave text reasoning with re-grounding on image features, reporting gains over its backbone on eight multimodal benchmarks.

Problem

Existing multimodal chain-of-thought methods reason entirely in text after a single visual encoding pass, so the model cannot revisit image regions during reasoning ("vision-blind reasoning").

Approach

Learnable control tokens (<Think>, <Look>, <Answer>) let the MLLM alternate between textual reasoning and reasoning-conditioned attention over visual tokens, with the attended features projected back into the context as implicit visual tokens, trained with SFT plus an auxiliary grounding loss.

Key results

• TVI-CoT reaches 64.3% on MMMU vs 58.2% for the Qwen3-VL-8B backbone (+6.1), and 60.2% on MathVerse vs 56.4% (+3.8).
• On MathVista TVI-CoT scores 76.6%, above Claude-Opus-4.1 (74.5%) and the backbone's 73.2%.
• Replacing learned grounding with random regions drops MMMU to 51.2%, and a fixed single interleaving step gives 62.3% on MMMU versus 64.3% for the adaptive design.

Surprising findings

• Random grounding regions drop MMMU accuracy to 51.2%, below the 58.2% text-only backbone, suggesting poor grounding actively hurts performance.
• Oracle human-annotated grounding regions improve results by only 3.0%, suggesting the learned grounding is already close to optimal.

Limitations

• Training data is largely synthetic, generated and validated by Gemini3-Pro and Qwen3-VL, which may propagate teacher biases.
• Grounding supervision for samples without annotations relies on weak signals from object detection or pretrained grounding models.
• The 32B model was trained only with LoRA due to memory constraints, so the larger-model results are not directly comparable to full fine-tuning.

Experimental setup

Datasets: LLaVA-178k Visual-CoT Zebra-CoT

Benchmarks: MMMU MMBench MathVerse MathVista ScienceQA MMStar AI2D MMT-Bench

Metrics: accuracy

Baselines: Qwen3-VL-8B LLaVA-CoT Insight-V Mulberry Vision-R1-7B VL-Rethinker-7B VAPO-Thinker-7B Visual-CoT GPT-5-high Gemini-2.5-Pro

Builds on: chain-of-thought Qwen3-VL Visual-CoT LLaVA-CoT

Closest papers by meaning

• Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision (ICLR 2026 · similarity 0.93)
• VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.92)
• Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning (NeurIPS 2025 · similarity 0.92)
• Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.92)
• UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs (NeurIPS 2026 · similarity 0.92)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.92)