Research map / Vision-language and multimodal / Visual question answering
CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CausalChaos!, a 4,945-sample causal Why-QA video dataset built from Tom and Jerry cartoons with multi-level answers, hard and causal-confusion distractors, and benchmarks of VideoQA and multimodal LLMs.
Problem
Existing causal video QA datasets require only surface-level understanding, have short causal chains, mostly single scenes, and easy distractors, so they fail to test deep causal reasoning across dynamic scenes.
Approach
Use animated Tom and Jerry cartoons, whose animation principles make cause-effect unambiguous, to build long causal-chain Why-QA pairs with two-level answers (primary answer and explanation), frequent scene changes, and hard and causally-confused distractor options generated via sentence-BERT mining and LLMs.
Key results
- Best MCQA accuracy under the answer-only protocol is GPT-4o at 63.64% (UD) and VILA at 77.22% (UD), versus 94.80% for humans and 20% chance.
- Under the stricter answer+explanation protocol, MIST reaches 44.88% and GPT-4o 48.17% (UD), against 4% chance.
- Adding CausalChaos! training data to NExT-QA improves HGA, MIST and VILA on the NExT-QA test set (e.g., VILA 67.23 to 68.46 MCQA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add causal-chain generation as an auxiliary task (ablation) | MIST · CausalChaos! UD, PS, and UN splits | accuracy: 62.22 -> 63.34; 44.88 -> 46.80; 62.22 -> 62.50; 42.86 -> 43.54; 55.91 -> 56.18; 37.90 -> 39.25 | Helped |
| Use causal-confusion incorrect options instead of vanilla hard negatives | MIST · CausalChaos! | accuracy: 62.22 -> 55.80 | Hurt |
| Use causal-confusion incorrect options instead of vanilla hard negatives | MIST-CC · CausalChaos! | accuracy: 63.34 -> 58.76 | Hurt |
| Use causal-confusion incorrect options instead of vanilla hard negatives | GPT-4o · CausalChaos! | accuracy: 63.64 -> 54.95 | Hurt |
| Add a visual-feature projector to GPT-2 (ablation) | BlindGPT-2 · CausalChaos! UD | CapsMIX: 6.6006 -> 6.7582 | Helped |
| Add CausalChaos! training data | BlindQA trained on NextQA · NextQA test | MCQA accuracy: 32.73 -> 32.91 | Helped |
| Add CausalChaos! training data | HGA trained on NextQA · NextQA test | MCQA accuracy: 41.23 -> 41.44 | Helped |
| Add CausalChaos! training data | MIST trained on NextQA · NextQA test | MCQA accuracy: 55.44 -> 55.96 | Helped |
| Add CausalChaos! training data | VILA trained on NextQA · NextQA test | MCQA accuracy: 67.23 -> 68.46 | Helped |
| Add CausalChaos! training data | HGA trained on NextQA · NextQA test | CapsMIX: 1.44 -> 1.48 | Helped |
| Add CausalChaos! training data | VILA trained on NextQA · NextQA test | CapsMIX: 2.66 -> 2.86 | Helped |
Surprising findings
- Performance drops substantially on the causal-confusion distractor set (e.g., MIST 62.22 to 55.80, GPT-4o 63.64 to 54.95), indicating shortcut reliance on object/action matching.
- Sentence-BERT similarity exceeds 90% between an action and its negated causal-confusion version, showing the embedding metric misses causal differences.
Limitations
- Videos are not distributed (only annotations), and the source is a single cartoon series with synthetic-style visuals.
- Causal chain length is measured by GPT-4o, with manual verification agreeing only 89%.
- Authors caution that models trained only on the dataset should not be deployed in real-world settings.
Experimental setup
Datasets: CausalChaos! NExT-QA
Benchmarks: CausalChaos! NExT-QA
Metrics: accuracy (A and A+E protocols) BLEU-1/2/3 METEOR ROUGE SPICE CIDEr Sentence-BERT CapsMIX
Baselines: BlindQA EVQA CoMem HME HCRN HGA MIST BLIP-2 Video-LLaMA VideoChat2 GPT-4o VILA
Builds on: MIST NExT-QA CausalVidQA IntentQA
Closest papers by meaning
- MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning (NeurIPS 2024 · similarity 0.86)
- Two Causally Related Needles in a Video Haystack (NeurIPS 2025 · similarity 0.85)
- Image Content Generation with Causal Reasoning (AAAI 2024 · similarity 0.85)
- From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.84)
- Cross-modal Causal Relation Alignment for Video Question Grounding (CVPR 2025 · similarity 0.84)
- Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.