Research map / Vision-language and multimodal / Visual question answering
Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a video-grounded entailment tree reasoning framework for commonsense VQA that decomposes answer statements, grounds them in video moments via captioning and retrieval, and verifies them with VLM probing, plus an LLM-based answer-set de-biasing procedure.
Problem
Video QA models with VLMs may rely on spurious correlations and textual/answer-set biases rather than genuine video-grounded commonsense reasoning, and their black-box decisions are hard to inspect.
Approach
Generalizes NLP entailment trees to video: answer options become statements that are recursively decomposed into sub-statements, each verified against a question-conditioned video moment grounded via fact-conditioned captioning, LLM-based anchor frame retrieval over structured triplets, and temporal navigation; a dynamic expansion prunes decompositions that do not raise confidence.
Key results
- Integrating the entailment tree framework improves every evaluated VLM by about 1-4% on average across NExT-QA, IntentQA and Video-MME (e.g., LLaVA-1.5 avg 37.1 to 41.5 on the original sets).
- On the de-biased NExT-QA set, VideoLLaVA rises from 41.4 to 47.0 with the framework (VideoLLaVA avg across de-biased sets 41.2 to 48.0 for the framework versus base models).
- Using VideoLLaVA as the prover, the framework is competitive with GPT-4-based VideoAgent/VideoTree/LLoVi on de-biased sets while using 257x fewer reasoner parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add video-grounded entailment tree reasoning | VideoLLaVA · NExT-QA (original), temporal and causal | accuracy: 56.0 -> 58.3; 60.4 -> 62.7 | Helped |
| Add video-grounded entailment tree reasoning | VideoChat2 · IntentQA (original), temporal | accuracy: 60.4 -> 62.3 | Helped |
| Add video-grounded entailment tree reasoning | VideoChat2 · IntentQA (original), causal | accuracy: 63.2 -> 63.8 | Helped |
| Add video-grounded entailment tree reasoning | VideoLLaVA · NExT-QA (rewritten) | accuracy: 51.1 -> 55.4 | Helped |
| Replace LLaMA-3-8B with GPT-4 for statement decomposition (ablation) | VideoLLaVA with entailment trees · NExT-QA | accuracy: 60.5 -> 61.6; 55.4 -> 56.1 | Helped |
| Add fact-conditioned captioning (ablation) | Video-grounded entailment tree reasoning · NExT-QA | accuracy: 56.2 -> 59.5; 49.6 -> 53.3 | Helped |
| Add structure-based retrieval (ablation) | Video-grounded entailment tree reasoning with fact-conditioned captioning · NExT-QA | accuracy: 59.5 -> 60.5; 53.3 -> 55.4 | Helped |
| Add fact-conditioned captioning (ablation) | Video-grounded entailment tree reasoning with structure-based retrieval · NExT-QA | accuracy: 58.4 -> 60.5; 52.7 -> 55.4 | Helped |
| Increase frame count from 4 to 24 (ablation) | Video-grounded entailment tree reasoning · NExT-QA | accuracy: 57.7 -> 60.7; 51.9 -> 55.4 | Helped |
| Increase frame count from 24 to 32 (ablation) | Video-grounded entailment tree reasoning · NExT-QA | accuracy: 60.7 -> 61.0; 55.4 -> 55.7 | Helped |
| Replace grounded video fragments with the full video (ablation) | Video-grounded entailment tree reasoning · NExT-QA | accuracy: 60.5 -> 58.3; 55.4 -> 51.7 | Hurt |
| Use dynamic rather than fixed-depth tree expansion (ablation) | Video-grounded entailment tree reasoning · NExT-QA | accuracy: 60.3 -> 60.5; 55.3 -> 55.4 | Helped |
Surprising findings
- Video-based VLMs drop 8-10% on the LLM-rewritten de-biased answer sets even though questions and correct answers are unchanged.
- Structure-guided retrieval alone slightly hurts performance (56.2 to 58.4 with fact captioning, but structure-only lowers results relative to fact-conditioned captioning alone).
Limitations
- Performance depends on grounding accuracy; ground-truth moments give further gains, showing headroom.
- Results depend on the captioner and LLM quality; larger LLMs give only marginal gains over LLaMA-3-8B.
- Evaluated mainly on multiple-choice VQA with accuracy metrics.
Experimental setup
Datasets: NExT-QA IntentQA Video-MME NExT-GQA
Benchmarks: NExT-QA IntentQA Video-MME
Metrics: accuracy
Baselines: BLIP-2 LLaVA-1.5 VideoChat2 VideoLLaVA VideoLLaMA VideoAgent VideoTree LLoVi
Builds on: EntailmentBank Entailer VideoAgent VideoTree LLoVi
Closest papers by meaning
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.92)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.91)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.90)
- Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering (CVPR 2024 · similarity 0.90)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.90)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.