Research map / Vision-language and multimodal / Visual question answering

Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models

CVPR 2024 · Vision-language and multimodal › Visual question answering · approach: Multimodal LLM question answering · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Causal-CoG, a training-free prompting/decoding strategy that makes an MLM generate image descriptions as context, uses a causal (NDE vs TIE) filter to decide when to use them, and aggregates candidate answers weighted by their indirect effect.

Problem

Multi-modal language models often give inaccurate or hallucinated answers to VQA questions; the paper asks whether self-generated image context can improve precise answers at inference time without training.

Approach

Generate multiple image descriptions with the MLM itself, answer questions conditioned on them, and use causal-effect estimates (natural direct effect vs total indirect effect, via JSD) to filter samples where context helps and to weight candidate answers by their TIE.

Key results

• On LLaVA, Causal-CoG improves accuracy by +6.30% on POPE, +13.69% on Vizwiz*, and +6.43% on VQAv2* over direct decoding.
• On LLaVA-v1.5, gains are smaller (e.g., +7.43% on Vizwiz*, +3.60% on VSR) and MME and GQA* slightly drop (-0.43, -0.87).
• Using TIE-weighted top-k aggregation outperforms likelihood-weighted, similarity-weighted, and unweighted aggregation, and removing the causality filter reduces accuracy on most benchmarks.

Measured effects reported in the paper

Surprising findings

• On LLaVA-v1.5, Causal-CoG slightly reduces MME (-0.43) and GQA* (-0.87) accuracy.
• Naive single-candidate context prompting (Naive-CoG) already improves LLaVA on most benchmarks, indicating the generated context itself helps.

Limitations

• Requires generating many candidate contexts per sample, increasing inference cost.
• Gains are inconsistent on the stronger LLaVA-v1.5 (slight drops on MME and GQA*) and weaker on cognition tasks.
• Evaluation uses reformed single-choice versions of several benchmarks rather than original formats.

Experimental setup

Datasets: MME SEEDBench MMBench POPE VSR Winoground (ReForm-Eval split) OKVQA (ReForm-Eval split) VQAv2 (ReForm-Eval split) Vizwiz (ReForm-Eval split) GQA (ReForm-Eval split)

Benchmarks: MME SEEDBench MMBench POPE VSR Winoground* OKVQA* VQAv2* Vizwiz* GQA*

Metrics: accuracy JSD-based NDE/TIE

Baselines: Ensemble (prompt ensemble) One-shot in-context learning Naive-CoG LLaVA direct decoding

Builds on: chain-of-thought self-consistency LLaVA Flamingo one-shot in-context learning

Closest papers by meaning

• SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.88)
• Understanding Information Storage and Transfer in Multi-Modal Large Language Models (NeurIPS 2024 · similarity 0.87)
• When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.87)
• Link-Context Learning for Multimodal LLMs (CVPR 2024 · similarity 0.87)
• Two Causally Related Needles in a Video Haystack (NeurIPS 2025 · similarity 0.86)
• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.86)