Research map / Large language models / Retrieval and knowledge
BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BDIQA, a synthetic VirtualHome-based VideoQA benchmark testing belief, desire, and intention theory-of-mind reasoning at two difficulty levels, and evaluates pre-trained and end-to-end VideoQA models.
Problem
Current VideoQA datasets focus on causal and temporal reasoning and rarely test human theory-of-mind (belief, desire, intention) reasoning in videos.
Approach
A two-level, animation-generated VideoQA benchmark grounded in children's ToM development, with joint BDI questions plus perception, where-location and yes/no questions; it also improves Frozen and JustAsk by swapping visual backbones and adding a memory module.
Key results
- Comem reaches the best supervised overall accuracy of 79.84%, versus 84.23% human accuracy on BDI questions; zero-shot JustAsk gets 13.55% BDI accuracy.
- Adding a memory module and RR visual features to JustAsk raises overall accuracy from 64.59% to 81.01%, surpassing Comem.
- Frozen zero-shot achieves only about 0.40% BDI accuracy, and VideoMAE-based desire classification is around 10%, below random.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace S3D with ResNet and ResNeXt-101 visual features (ablation) | Frozen · BDIQA supervised | overall accuracy: 71.49% -> 77.81% | Helped |
| replace S3D with ResNet and ResNeXt-101 visual features (ablation) | Frozen · BDIQA supervised | Acc_bdi: 70.03% -> 79.13% | Helped |
| add memory module (ablation) | Frozen · BDIQA supervised | overall accuracy: 71.49% -> 72.22% | Helped |
| add memory module to the model with RR visual features (ablation) | Frozen + RR · BDIQA supervised | overall accuracy: 77.81% -> 78.62% | Helped |
| replace S3D with ResNet and ResNeXt-101 visual features (ablation) | JustAsk · BDIQA supervised | overall accuracy: 64.59% -> 78.83% | Helped |
| add memory module to the model with RR visual features (ablation) | JustAsk + RR · BDIQA supervised | overall accuracy: 78.83% -> 81.01% | Helped |
| replace S3D with ResNet and ResNeXt-101 visual features (ablation) | Frozen · BDIQA supervised | overall accuracy: 71.49% -> 77.84% | Helped |
| use appearance and motion features together rather than a single visual feature (ablation) | Frozen · BDIQA supervised | Acc_bdi: 79.13% | Helped |
| train with 10% data rather than zero-shot | ClipBERT · BDIQA | Acc_bdi: 1.09% -> 32.14% | Helped |
| train with 10% data rather than zero-shot | Frozen · BDIQA | Acc_bdi: 0.40% -> 5.04% | Helped |
| train with 10% data rather than zero-shot | JustAsk · BDIQA | Acc_bdi: 13.55% -> 16.72% | Helped |
Surprising findings
- Pre-trained VideoQA models such as JustAsk and Frozen perform poorly after fine-tuning on BDIQA despite large-scale VideoQA pretraining.
- Models answer belief questions better than the simpler where-location questions they are supposed to depend on, contrary to the multi-step design.
Limitations
- Benchmark is small (3,527 videos) and synthetic animations with simple 320x240 frames and templated questions.
- Only 10 household activities and two characters, limiting diversity of questions and answers.
- Memory module gains over the baselines are modest (overall 1.17% over the baseline per the text).
Experimental setup
Datasets: BDIQA
Benchmarks: BDIQA
Metrics: accuracy overall Acc Acc_bdi Acc_p
Baselines: HME Comem HCRN HGA Dual Frozen JustAsk ClipBERT Uniformer VideoMAE VideoMAEV2 XClip
Builds on: Frozen JustAsk HME Comem HCRN HGA VirtualHome
Closest papers by meaning
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
- Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation (CVPR 2025 · similarity 0.88)
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.88)
- HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes (ICLR 2026 · similarity 0.88)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.