Research map / Vision-language and multimodal / Multimodal LLMs

What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces C-VQA, a benchmark of real (VQAv2-based) and synthetic VQA questions with counterfactual presuppositions, and shows that multimodal LLMs and neuro-symbolic models drop sharply on counterfactual versions.

Problem

Existing VQA benchmarks do not test whether multimodal LLMs can perform counterfactual "what if" reasoning over image content; prior counterfactual VQA work (VQA-CP, CF-VQA) instead targets train/test distribution shift.

Approach

Augment VQAv2 questions with counterfactual presuppositions (numerical direct, numerical indirect, boolean) generated by ChatGPT via chain-of-thought and in-context prompting, verified manually, plus procedurally generated synthetic flower- and dot-counting puzzles with counterfactual multiple-choice questions.

Key results

• Qwen-VL-Chat drops from 65.1% to 30.7% on numerical direct and from 69.2% to 29.4% on numerical indirect questions when counterfactual presuppositions are added.
• GPT-4V (on a 10% subset) drops 35.8 points on numerical direct (50.4 to 14.6) and 17.9 points on boolean (65.9 to 48.0).
• On C-VQA-Synthetic, InstructBLIP (Vicuna-7B) scores about 26.9% original and 25.7% counterfactual, near the 25% random level, while other models perform at or below chance.

Measured effects reported in the paper

Surprising findings

• Chain-of-thought prompting ('Let's think step by step') leaves performance constant or lowers it on counterfactual questions.
• Several end-to-end models answer counterfactual questions correctly but fail the original questions (e.g., LLaVA-13B on numerical direct).

Limitations

• Small number of images (about 3K real and 3K synthetic), limiting coverage.
• Evaluation of GPT-4V limited to a 10% random subset due to rate limits.
• Synthetic questions are out-of-distribution, so many models score near chance, making them less diagnostic.

Experimental setup

Datasets: VQAv2 COCO Icons-50

Benchmarks: C-VQA-Real C-VQA-Synthetic

Metrics: accuracy

Baselines: ViperGPT VisProg BLIP2 InstructBLIP MiniGPT-4 MiniGPT-v2 LLaVA-1.5 CogVLM Qwen-VL GPT-4V

Builds on: VQAv2 CLEVR ViperGPT VisProg chain-of-thought prompting

Closest papers by meaning

• Counterfactual reasoning: an analysis of in-context emergence (NeurIPS 2025 · similarity 0.91)
• Reasoning Elicitation in Language Models via Counterfactual Feedback (ICLR 2025 · similarity 0.89)
• CF-VLM：CounterFactual Vision-Language Fine-tuning (NeurIPS 2025 · similarity 0.89)
• CFPO: Counterfactual Policy Optimization for Multimodal Reasoning (ICML 2026 · similarity 0.89)
• V-CECE: Visual Counterfactual Explanations via Conceptual Edits (NeurIPS 2025 · similarity 0.88)
• Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments (AAAI 2025 · similarity 0.88)