Research map / Video / Video-language understanding
YTCommentQA: Video Question Answerability in Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces YTCommentQA, a dataset of real YouTube viewer questions from instructional videos labeled for answerability and required modality (visual, script, or both), and benchmarks LLMs and a multimodal model on answerability classification.
Problem
Video QA models are trained on questions written from video content, but real users often ask questions the video cannot answer; determining whether an instructional video can answer a question, and which modality is needed, is understudied.
Approach
Collect naturally occurring YouTube comment questions whose replies contain timestamps, then annotate via crowd workers whether the answer lies in the visual snippet, the script snippet, both, or neither, and benchmark segment-level and video-level answerability classification.
Key results
- Segment answerability: Llama-2 (13B) reached 55.49 F1, versus 33.02 for GPT-4 and 46.55 for SeViLA.
- Video answerability (5-way): Llama-2 (13B) reached 37.70 accuracy, versus 27.03 for GPT-4 and 35.27 for SeViLA.
- Combined-answerable questions were mostly mispredicted: about 85% of them were wrong for GPT-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the fine-tuned Llama-2 size from 7B to 13B | Llama-2 on YTCommentQA Segment Answerability Classification · YTCommentQA Segment Answerability Classification | F1 score: 45.78 -> 55.49 | Helped |
| Use GPT-4 instead of GPT-3.5 | Zero-shot LMs on YTCommentQA Segment Answerability Classification · YTCommentQA Segment Answerability Classification | F1 score: 31.16 -> 33.02 | Helped |
| Increase the fine-tuned Llama-2 size from 7B to 13B | Llama-2 on YTCommentQA Video Answerability Classification · YTCommentQA Video Answerability Classification | accuracy: 36.24 -> 37.70 | Helped |
| Use GPT-4 instead of GPT-3.5 | Zero-shot LMs on YTCommentQA Video Answerability Classification · YTCommentQA Video Answerability Classification | accuracy: 18.42 -> 27.03 | Helped |
Surprising findings
- GPT-4 often labeled visually answerable questions as unanswerable, with 53% of its false cases from this confusion.
- A question-prefix frequency baseline scored lower than the fine-tuned models, suggesting little answerability bias by question type.
Limitations
- Annotations focus on segments around timestamps, so the full video context may be missed.
- Video inputs were reduced to summaries or partial chunks due to input length limits.
Experimental setup
Datasets: YT-Temporal-180M HowTo100M
Benchmarks: YTCommentQA
Metrics: F1 score accuracy
Baselines: Llama-2 (7B) Llama-2 (13B) GPT-3.5 GPT-4 SeViLA
Builds on: HowTo100M YT-Temporal-180M SeViLA
Closest papers by meaning
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.86)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.86)
- Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering (CVPR 2024 · similarity 0.86)
- Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models (ICLR 2025 · similarity 0.86)
- Track the Answer: Extending TextVQA from Image to Video with Spatio-Temporal Clues (AAAI 2025 · similarity 0.86)
- Understanding Complexity in VideoQA via Visual Program Generation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.