Research map / Vision-language and multimodal / Multimodal LLMs
Revealing Multimodal Causality with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLLM-CD, a framework that uses multimodal LLMs for contrastive factor discovery from unstructured multimodal data, FCI-based structure discovery, and iterative counterfactual reasoning to refine causal graphs, evaluated on a synthetic apple dataset and a lung cancer dataset.
Problem
Causal discovery from multimodal unstructured data requires identifying latent causal factors across modalities and resolving structural ambiguities (Markov equivalence) from purely observational data, which existing LLM-based methods like COAT do not handle.
Approach
Uses intra- and inter-modal contrastive sample pairs to prompt an MLLM to discover causal factors, applies a statistical CD algorithm (FCI) to the annotated data, and then generates validated multimodal counterfactual samples from MLLM world knowledge to iteratively refine the graph.
Key results
- On the Lung Cancer dataset, averaged over four MLLMs, MLLM-CD reaches node F1 0.90 and adjacency F1 0.68 with ESHD 5.33, versus 0.52 / 0.32 / 9.42 for COAT.
- On the MAG dataset with Gemini 2.0, MLLM-CD achieves node F1 0.87 and adjacency F1 0.60 with ESHD 14.00, compared with 0.51 and 0.37 for COAT on adjacency F1 with ESHD 16.00.
- Ablation on MAG shows removing counterfactual reasoning lowers adjacency F1 from 0.60 to 0.52, and removing contrastive factor discovery lowers node F1 from 0.87 to 0.73.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MLLM-CD instead of the strongest listed baseline, COAT | MAG (average across LLMs) · MAG | NF: 0.53 ± 0.11 -> 0.89 ± 0.06 | Helped |
| Use MLLM-CD instead of the strongest listed baseline, META | MAG (average across LLMs) · MAG | AF: 0.46 ± 0.07 -> 0.57 ± 0.06 | Helped |
| Use MLLM-CD instead of the strongest listed baseline, COAT | MAG (average across LLMs) · MAG | ESHD: 16.42 ± 2.02 -> 13.42 ± 2.68 | Helped |
| Use MLLM-CD instead of the strongest listed baseline, COAT | Lung Cancer (average across LLMs) · Lung Cancer | NF: 0.52 ± 0.13 -> 0.90 ± 0.07 | Helped |
| Use MLLM-CD instead of the strongest listed baseline, META | Lung Cancer (average across LLMs) · Lung Cancer | AF: 0.59 ± 0.17 -> 0.68 ± 0.19 | Helped |
| Use MLLM-CD instead of the strongest listed baseline, COAT | Lung Cancer (average across LLMs) · Lung Cancer | ESHD: 9.42 ± 2.31 -> 5.33 ± 0.78 | Helped |
| Add contrastive factor discovery (ablation) | MLLM-CD · MAG | NF: 0.73 ± 0.07 -> 0.87 ± 0.03 | Helped |
| Add contrastive factor discovery (ablation) | MLLM-CD · Lung Cancer | NF: 0.62 ± 0.04 -> 0.97 ± 0.05 | Helped |
| Add counterfactual reasoning (ablation) | MLLM-CD · MAG | AF: 0.52 ± 0.06 -> 0.60 ± 0.06 | Helped |
| Add counterfactual reasoning (ablation) | MLLM-CD · Lung Cancer | AF: 0.38 ± 0.04 -> 0.87 ± 0.13 | Helped |
| Combine intra-modal and inter-modal contrastive sampling instead of using intra-modal sampling alone (ablation) | MLLM-CD factor discovery · MAG | NF: 0.82 ± 0.12 -> 0.87 ± 0.03 | Helped |
| Use inter-modal-only contrastive sampling instead of the combined strategy (ablation) | MLLM-CD factor discovery · MAG | NF: 0.87 ± 0.03 -> 0.63 ± 0.18 | Hurt |
Surprising findings
- Baseline META achieves high adjacency precision (e.g., 1.00 on Lung Cancer with GPT-4o) despite poor node precision, suggesting LLM-only causal graphs are certain but overly complex.
- Inter-modal contrastive sampling alone yields lower quantitative results than intra-modal, serving only as a complement rather than a standalone factor discovery strategy.
Limitations
- Datasets are small (200 synthetic and 60 real-world samples) and ground truth graphs rely on expert knowledge.
- Only text and image modalities are covered; the approach is bounded by the core MLLM's modality support.
- MLLM hallucination and bias may affect factor discovery and counterfactual validity.
Experimental setup
Datasets: Multimodal Apple Gastronome (MAG) Lung Cancer (MedPix)
Benchmarks: MAG Lung Cancer
Metrics: Node Precision Node Recall Node F1-score Adjacency Precision Adjacency Recall Adjacency F1-score ESHD
Baselines: META Pairwise Triplet COAT
Builds on: COAT FCI META Pairwise prompting Triplet prompting
Closest papers by meaning
- CausalTab: Pretraining Across Causal Environments for Tabular Causal Discovery (NeurIPS 2026 · similarity 0.90)
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.88)
- End-To-End Causal Effect Estimation from Unstructured Natural Language Data (NeurIPS 2024 · similarity 0.87)
- Discovery of the Hidden World with Large Language Models (NeurIPS 2024 · similarity 0.87)
- Differentiable Causal Discovery for Latent Hierarchical Causal Models (ICLR 2025 · similarity 0.86)
- Discovering Mixtures of Structural Causal Models from Time Series Data (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.