Research map / Vision-language and multimodal / Multimodal LLMs

CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CoMT, a benchmark of 3,853 samples requiring multi-modal (image+text) reasoning outputs across four visual operations, and finds that LVLMs perform near random while multimodal in-context demonstrations help.

Problem

Existing multimodal chain-of-thought benchmarks take multi-modal input but only produce text-only reasoning, missing visual operations and leading to vague expressions; the paper asks whether LVLMs can reason by generating interleaved image-text thoughts.

Approach

A benchmark whose rationales interleave generated images with text (chain of multi-modal thought), covering visual creation, deletion, update, and selection, built by templating and annotating existing datasets (GeoQA+, JHU-CROWD++, KILOGRAM, spot-diff) and evaluating prompting strategies including multimodal in-context learning.

Key results

• Gemini achieves the best average accuracy of 33.44% with Direct prompting (28.67 Macro-F1), only marginally above the 25.48% random baseline; most open LVLMs are at or near chance.
• VoT (textual visualization) lowers performance on average by about 12.28% relative to direct prompting across the evaluated models, per the paper's claim.
• Multimodal in-context demonstrations (interleaved image and text rationales) improve performance over zero-shot and text-only ICL, with performance declining beyond four demonstrations.

Measured effects reported in the paper

Surprising findings

• Text-only visualization prompting (VoT) degrades performance for nearly all models rather than helping.
• IDEFICS2-8B, with fine-grained multimodal alignment, reportedly surpasses 13B models and approaches Gemini-Pro, suggesting alignment quality matters more than parameter count.

Limitations

• Small benchmark of 3,853 samples built from four existing datasets with templated rationales.
• Most evaluated models are near random chance, and the paper's evaluated models include few with genuine image generation abilities.
• Human image-quality judgments were made on a sampled subset (50 per task per model).

Experimental setup

Datasets: GeoQA+ JHU-CROWD++ KILOGRAM spot-diff puzzles (allstarpuzzles.com) M3CoT

Benchmarks: CoMT

Metrics: accuracy Macro-F1 CLIPScore ROSCOE

Baselines: Direct CoT Desp-CoT VoT Random

Builds on: Multimodal-CoT Visual Sketchpad-style visual thought (VoT) M3CoT GeoQA+ JHU-CROWD++ KILOGRAM

Closest papers by meaning

• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.92)
• RBench-V: A Primary Assessment for Visual Reasoning Models with Multimodal Outputs (NeurIPS 2025 · similarity 0.91)
• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.91)
• Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
• Interleaved-Modal Chain-of-Thought (CVPR 2025 · similarity 0.90)
• Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision (ICLR 2026 · similarity 0.90)