Research map / Vision-language and multimodal / Visual question answering
Can I Trust Your Answer? Visually Grounded Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NExT-GQA, a weakly-supervised grounded VideoQA benchmark extending NExT-QA with 10.5K temporal grounding labels, shows that strong VLMs answer correctly without grounding their answers, and proposes a Gaussian-mask grounding module with cross-modal contrastive learning (NG+).
Problem
Current VideoQA vision-language models give strong answer accuracy, but it is unclear whether their answers are grounded in the relevant video moments rather than language shortcuts or irrelevant visual context.
Approach
Extend VideoQA to weakly-supervised grounded VideoQA: models must answer and output a temporal evidence window trained only with QA supervision. A learnable Gaussian temporal mask is added to existing VLMs, plus an auxiliary cross-modal question-video contrastive objective (NG+) to make the grounding question-relevant.
Key results
- Best QA models reach about 69-70% QA accuracy but only about 16-17% grounded QA accuracy (Acc@GQA) on NExT-GQA test, versus 93% human QA accuracy with 82% grounded.
- NG+ improves FrozenBiLM from 69.1 to 70.8 Acc@QA and Temp[CLIP] from 59.4 to 60.2, with IoP@0.5 gains of up to 3.2 points.
- Human study yields 93.3% QA accuracy and 82.1% grounded QA accuracy, showing a large gap to models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Gaussian mask optimization and cross-modal self-supervision (ablation) | FrozenBiLM · NExT-GQA test | Acc@QA: +1.7 | Helped |
| Add Gaussian mask optimization and cross-modal self-supervision (ablation) | FrozenBiLM · NExT-GQA test | Acc@GQA: +1.7 | Helped |
| Add Gaussian mask optimization and cross-modal self-supervision (ablation) | FrozenBiLM · NExT-GQA test | IoP@0.5: +1.6 | Helped |
| Add Gaussian mask optimization and cross-modal self-supervision (ablation) | Temp[CLIP] · NExT-GQA test | Acc@QA: +0.8 | Helped |
| Add Gaussian mask optimization and cross-modal self-supervision (ablation) | Temp[CLIP] · NExT-GQA test | IoU@0.5: +4.8 | Helped |
| Use ground-truth temporal segments instead of normal uniform frame sampling | Temp[CLIP] · NExT-GQA test | Acc@QA: 59.4 -> 59.8 | No clear effect |
| Use ground-truth temporal segments instead of normal uniform frame sampling | FrozenBiLM · NExT-GQA test | Acc@QA: 69.1 -> 68.5 | Hurt |
| Exclude ground-truth temporal segments from video frames | Temp[CLIP] · NExT-GQA test | Acc@QA: 59.4 -> 59.1 | No clear effect |
| Exclude ground-truth temporal segments from video frames | FrozenBiLM · NExT-GQA test | Acc@QA: 69.1 -> 68.2 | No clear effect |
Surprising findings
- A BlindQA language-only model reaches about 80% of the standard VQA performance, indicating strong language shortcuts.
- Providing ground-truth temporal segments (PosQA) yields only marginal gains and can hurt stacked-style transformers, while excluding them (NegQA) barely degrades performance.
Limitations
- NG+ demands more memory and training time.
- Analyses are restricted to multi-choice QA.
- A large gap to human performance remains in both QA and grounding.
Experimental setup
Datasets: NExT-QA NExT-GQA
Benchmarks: NExT-GQA NExT-QA
Metrics: accuracy Acc@GQA IoP IoU mIoP mIoU
Baselines: VGT VIOLETv2 Temp[Swin] Temp[CLIP] Temp[BLIP] FrozenBiLM IGV SeViLA BlindQA
Builds on: NExT-QA FrozenBiLM VGT VIOLETv2 BLIP-2
Closest papers by meaning
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (CVPR 2018 · similarity 0.90)
- Visual7W: Grounded Question Answering in Images (CVPR 2016 · similarity 0.90)
- Invariant Grounding for Video Question Answering (CVPR 2022 · similarity 0.90)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.89)
- Right this way: Can VLMs Guide Us to See More to Answer Questions? (NeurIPS 2024 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.