Research map / Vision-language and multimodal / Human and embodied modeling
Multimodal Event Causality Reasoning with Scene Graph Enhanced Interaction Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEIN, a scene-graph-enhanced GCN with optimal-transport object alignment across image pairs and cross-modal fusion, for multimodal event causality reasoning on Vis-Causal.
Problem
Multimodal event causality reasoning requires recognizing causal relations between two events given paired images, but existing methods ignore inter-object relations within an image and object interactions across the image pair.
Approach
Builds scene graphs per image and models them with a GCN, then aligns objects across the two time-consecutive images via entropic/IPOT optimal transport, using the transport cost matrix to fuse object features, combined with BERT text features through multi-head cross-modal attention and an adaptive text/multimodal prediction fusion.
Key results
- SEIN reaches overall R@1 18.09, R@5 53.19, R@10 73.40 on Vis-Causal, outperforming OARNet (17.38, 50.00, 71.28).
- Ablation: removing scene graph drops R@1 from 18.09 to 16.35; removing OT alignment drops R@1 to 17.21.
- Best performance with two GCN layers; performance drops quickly with more layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEIN instead of OARNet | Vis-Causal · Vis-Causal test | R@1: 17.38 -> 18.09 | Helped |
| Use SEIN instead of OARNet | Vis-Causal · Vis-Causal test | R@5: 50.00 -> 53.19 | Helped |
| Use SEIN instead of OARNet | Vis-Causal · Vis-Causal test | R@10: 71.28 -> 73.40 | Helped |
| add scene graph (ablation) | SEIN · Vis-Causal test | R@1: 16.35 -> 18.09 | Helped |
| add scene graph (ablation) | SEIN · Vis-Causal test | R@5: 51.22 -> 53.19 | Helped |
| add scene graph (ablation) | SEIN · Vis-Causal test | R@10: 71.63 -> 73.40 | Helped |
| add optimal transport-based alignment (ablation) | SEIN · Vis-Causal test | R@1: 17.21 -> 18.09 | Helped |
| add optimal transport-based alignment (ablation) | SEIN · Vis-Causal test | R@5: 51.87 -> 53.19 | Helped |
| add optimal transport-based alignment (ablation) | SEIN · Vis-Causal test | R@10: 71.92 -> 73.40 | Helped |
| add cross-modal fusion (ablation) | SEIN · Vis-Causal test | R@1: 17.62 -> 18.09 | Helped |
| add cross-modal fusion (ablation) | SEIN · Vis-Causal test | R@5: 52.11 -> 53.19 | Helped |
| add cross-modal fusion (ablation) | SEIN · Vis-Causal test | R@10: 72.24 -> 73.40 | Helped |
Limitations
- Relies on off-the-shelf Faster R-CNN and scene graph diagnosis toolkit, so scene graph quality bounds performance.
- Evaluated only on the Vis-Causal dataset in the main experiments.
- Improvements over OARNet are modest on R@1 (18.09 vs 17.38) and the Eating/Care subsets are small.
Experimental setup
Datasets: Vis-Causal ATOMIC
Benchmarks: Vis-Causal
Metrics: R@1 R@5 R@10
Baselines: Random BERT VCC iReason OARNet
Builds on: BERT Faster R-CNN IPOT GCN OARNet
Closest papers by meaning
- From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.87)
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.86)
- Asynchronous Collaborative Graph Representation for Frames and Events (CVPR 2025 · similarity 0.86)
- HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.85)
- Causal Inference over Visual-Semantic-Aligned Graph for Image Classification (AAAI 2025 · similarity 0.85)
- MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.