Research map / Video / Video-language understanding
MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Multi-Event Causal Discovery (MECD) task and dataset, and VGCM, a Granger-causality-inspired two-stream masked event prediction model with front-door adjustment and counterfactual inference for event-level causal diagrams in long videos.
Problem
Existing video causal reasoning is QA-based and limited to short videos with single or few events, lacking structured event-level causal diagrams for long multi-event videos.
Approach
Applies the Granger causality test to video events by comparing predictions of the result event with and without a premise event masked, then corrects causality confounding (front-door adjustment with chain-of-thought) and illusory causality (counterfactual intervention with existence-only descriptions).
Key results
- VGCM reaches 71.2% accuracy on MECD, outperforming GPT-4o (65.5%) by 5.7% and fine-tuned VideoLLaVA (67.1%) by 4.1%.
- Without causal inference, VGCM reaches 66.9% accuracy; front-door adjustment and counterfactual intervention each add gains in ablation.
- Inference takes 0.76 s per sample, 3-6x faster than video LLMs, with 8.57% overhead over VideoBERT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VGCM instead of fine-tuned VideoLLaVA | MECD causal discovery · MECD test | accuracy: 67.1 -> 71.2 | Helped |
| use VGCM instead of GPT-4o | MECD causal discovery · MECD test | accuracy: 65.5 -> 71.2 | Helped |
| use VGCM instead of GPT-4o | MECD causal discovery · MECD test | Ave SHD: 4.69 -> 4.19 | Helped |
| add causal inference (ablation) | VGCM · MECD test | accuracy: 67.0 -> 71.2 | Helped |
| add caption loss (ablation) | VGCM with visual reconstruction and similarity losses · MECD test | accuracy: 64.8 -> 67.0 | Helped |
| add visual reconstruction loss (ablation) | VGCM with caption and similarity losses · MECD test | accuracy: 64.8 -> 67.0 | Helped |
| add semantics similarity loss (ablation) | VGCM with caption and visual reconstruction losses · MECD test | accuracy: 65.3 -> 67.0 | Helped |
| add front-door adjustment (ablation) | VGCM with all three base losses · MECD test | accuracy: 67.0 -> 68.7 | Helped |
| add counterfactual intervention (ablation) | VGCM with all three base losses · MECD test | accuracy: 67.0 -> 69.3 | Helped |
| add both front-door adjustment and counterfactual intervention (ablation) | VGCM with all three base losses · MECD test | accuracy: 67.0 -> 71.2 | Helped |
| add counterfactual intervention (ablation) | VGCM · Illusory existence causality experiment | similarity division: 1.04 -> 0.93 | Helped |
| add front-door adjustment (ablation) | VGCM · Illusory temporal causality experiment | r0 accuracy: 63.6 -> 68.0 | Helped |
Surprising findings
- GPT-4 with few-shot prompting reaches only 59.6% accuracy, below several video LLMs, and adding more few-shot examples does not improve it much.
- Human deductive reasoning scores 87.2%, indicating a remaining gap to human performance.
Limitations
- Requires videos with timestamps and annotated event segments as input.
- Struggles with higher-level semantics such as obscure mental or emotional influences.
- Supervised paradigm only; unsupervised causal discovery not addressed.
Experimental setup
Datasets: ActivityNet Captions MECD
Benchmarks: MECD
Metrics: accuracy Structural Hamming Distance (SHD)
Baselines: GPT-4 GPT-4o Gemini-1.5-Pro VideoLLaVA VideoChat2 MiniGPT-4 MiniGPT4-video Video-llama VAR VideoBERT CLIP (ViT-L/14)
Builds on: VideoBERT VAR Granger causality front-door adjustment chain-of-thought prompting
Closest papers by meaning
- Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs (NeurIPS 2026 · similarity 0.87)
- Two Causally Related Needles in a Video Haystack (NeurIPS 2025 · similarity 0.86)
- CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes (NeurIPS 2024 · similarity 0.86)
- From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.85)
- EventLens: Event-Structure Reinforcement Learning for Video Understanding (NeurIPS 2026 · similarity 0.85)
- OpenCoF: Learning to Reason Through Video Generation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.