Research map / Vision-language and multimodal / Multimodal LLMs
Interleaved-Modal Chain-of-Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Interleaved-modal Chain-of-Thought (ICoT) with Attention-driven Selection (ADS), a training-free plug-and-play method that inserts attention-selected image patch tokens into VLM rationales, evaluated on Chameleon-7B and Qwen2-VL-7B.
Problem
Text-only chain-of-thought rationales from vision-language models struggle to express fine-grained associations with the input image, and current VLMs cannot generate interleaved image-text reasoning steps.
Approach
Instead of generating images, ADS triggers on a signal token (newline) and selects top-n visual tokens from the input image using the averaged attention map between the signal token and visual tokens, then re-inserts them in relative spatial order into the autoregressive generation, yielding interleaved visual-textual rationales without parameterization.
Key results
- ICoT on Chameleon-7B 1-shot reaches 32.3 on M3CoT, 53.4 on ScienceQA and 27.6 ROUGE-L on LLaVA-W, versus 28.4/48.5/23.9 for No-CoT.
- ICoT on Qwen2-VL-7B 0-shot improves LLaVA-W ROUGE-L from 32.7 (No-CoT) to 34.2, with the paper reporting gains up to 14% over the best baseline.
- Copying KV cache of selected visual tokens instead of inserting patches gives slight degradation (e.g., M3CoT 0-shot 29.1 vs 29.8) but reduces computation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use interleaved-modal CoT with ADS instead of the strongest listed baseline | Chameleon-7B · LLaVA-W 1-shot | ROUGE-L: 24.7 -> 27.6 | Helped |
| Use interleaved-modal CoT with ADS instead of the strongest listed baseline | Qwen2-VL-7B · LLaVA-W 1-shot | ROUGE-L: 33.9 -> 35.7 | Helped |
| Add ADS to generate interleaved-modal rather than text-only rationales (ablation) | ICoT on Chameleon-7B · M^3CoT 1-shot | accuracy: 29.2 -> 32.3 | Helped |
| Add ADS to generate interleaved-modal rather than text-only rationales (ablation) | ICoT on Chameleon-7B · LLaVA-W 1-shot | ROUGE-L: 24.5 -> 27.6 | Helped |
| Add fine-grained visual information to demonstrations instead of randomly sampled patches (ablation) | ICoT on Chameleon-7B · M^3CoT 1-shot | accuracy: 30.6 -> 32.3 | Helped |
| Add fine-grained visual information to demonstrations instead of randomly sampled patches (ablation) | ICoT on Chameleon-7B · LLaVA-W 1-shot | ROUGE-L: 25.9 -> 27.6 | Helped |
| Copy selected visual-token KV cache instead of inserting selected patches (ablation) | ICoT on Chameleon-7B · M^3CoT 1-shot | accuracy: 31.5 -> 32.3 | Hurt |
| Copy selected visual-token KV cache instead of inserting selected patches (ablation) | ICoT on Chameleon-7B · ScienceQA 1-shot | accuracy: 52.9 -> 53.4 | Hurt |
| Copy selected visual-token KV cache instead of inserting selected patches (ablation) | ICoT on Chameleon-7B · LLaVA-W 1-shot | ROUGE-L: 27.0 -> 27.6 | Hurt |
| Use model-written rather than human-written demonstrations (ablation) | ICoT on Chameleon-7B · M^3CoT 1-shot | accuracy: 32.3 -> 31.5 | Hurt |
| Use model-written rather than human-written demonstrations (ablation) | ICoT on Chameleon-7B · ScienceQA 1-shot | accuracy: 53.4 -> 51.8 | Hurt |
| Use model-written rather than human-written demonstrations (ablation) | ICoT on Chameleon-7B · LLaVA-W 1-shot | ROUGE-L: 27.6 -> 26.7 | Hurt |
Surprising findings
- Model-written demonstrations (VLM generates its own visual rationales) underperform manually designed fine-grained demonstrations.
- Multimodal CoT with text-only rationales sometimes underperforms No-CoT on Qwen2-VL and Chameleon in some settings.
Limitations
- ADS depends on a signal token; frequent generation of the trigger degrades quality.
- Fixed number of selected patches is sub-optimal and adds memory overhead for storing attention scores.
- Gains are modest on some benchmarks and the method relies on manually designed demonstrations.
Experimental setup
Datasets: M3CoT ScienceQA LLaVA-Bench In-the-Wild Flickr30k OKVQA
Benchmarks: M3CoT ScienceQA LLaVA-Bench In-the-Wild Flickr30k OKVQA
Metrics: accuracy ROUGE-L CIDEr VQA-ACC
Baselines: No-CoT Multimodal CoT CCoT DDCoT SCAFFOLD
Builds on: chain-of-thought prompting multimodal chain-of-thought DDCoT CCoT SCAFFOLD
Closest papers by meaning
- Let’s Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts (AAAI 2026 · similarity 0.91)
- Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning (AAAI 2024 · similarity 0.91)
- Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought (CVPR 2026 · similarity 0.91)
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.91)
- Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.91)
- CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.