Research map / Vision-language and multimodal / Multimodal LLMs

Compositional Chain-of-Thought Prompting for Large Multimodal Models

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 92 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Compositional Chain-of-Thought (CCoT), a zero-shot prompting method where an LMM first generates a JSON scene graph from the image and task, then answers conditioned on it, improving compositional and general multimodal benchmarks without finetuning.

Problem

LMMs treat images as a "bag of objects" and struggle with attributes and inter-object relations; scene graph supervision is costly and finetuning on SG data causes catastrophic forgetting.

Approach

Use the LMM itself to generate a task-conditioned scene graph (objects, attributes, relationships in JSON) as an intermediate chain-of-thought step, then prompt the same LMM with image, scene graph, and question to answer, with no finetuning or annotated SGs.

Key results

• GPT-4V-CCoT improves Winoground Group score from 33.5 to 41.0 (ZS-CoT 41.0 vs CCoT 43.3 on WHOOPS!; Winoground group 33.5 to 43.3 on WHOOPS! row) and beats prior SGVL SOTA without finetuning.
• LLaVA-1.5-13B-CCoT reaches 69.7 on SEED-Bench-Image vs 68.2 baseline, and 70.7 on MMBench vs 67.0, with WHOOPS! VQA rising from 47.3 to 61.2.
• ZS-CoT often degrades performance on compositional and multimodal benchmarks, e.g., InstructBLIP-13B SEED-I drops from 48.2 to 37.6.

Measured effects reported in the paper

Surprising findings

• Zero-shot CoT ('Let's think step-by-step') frequently hurts LMM performance on compositional and multimodal benchmarks, attributed to being designed for LLMs without visual grounding.
• Removing the image in the response step drops accuracy by 26.1% on SEED-Bench, and random scene graphs reduce performance by 5.6%, suggesting gains require both image and accurate SG content.

Limitations

• Limited by the LLM context length, since generated scene graphs and images both consume tokens.
• Scene graphs are not useful for tasks emphasizing language over visual reasoning, such as document understanding.
• Gains are small or absent for the 7B model (LLaVA-1.5-7B-CCoT shows no noticeable gain on SEED-Bench).

Experimental setup

Datasets: Winoground WHOOPS! SEED-Bench MMBench LLaVA-Bench (In-the-Wild) GQA VL-Checklist

Benchmarks: Winoground WHOOPS! SEED-Bench MMBench LLaVA-Bench (In-the-Wild)

Metrics: accuracy text score image score group score weighted average accuracy

Baselines: Zero-shot CoT (ZS-CoT) SGVL MMCoT DDCoT VidIL CLIP BLIP BLIP2 mPlug-OWL2 QwenVL-Chat

Builds on: zero-shot chain-of-thought Multimodal-CoT DDCoT VidIL

Closest papers by meaning

• Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.91)
• Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
• Interleaved-Modal Chain-of-Thought (CVPR 2025 · similarity 0.89)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
• Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning (AAAI 2024 · similarity 0.89)
• Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual Prompting (AAAI 2026 · similarity 0.89)