Research map / Video / Video-language understanding
FriendsQA: A New Large-Scale Deep Video Understanding Dataset with Fine-grained Topic Categorization for Story Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StoryMind, an LLM-based multi-agent (generator plus two reviewers) framework that automatically builds FriendsQA, a 44.6K-question deep video understanding dataset from Friends with 14 fine-grained story topics, and benchmarks 10 VideoQA models on it.
Problem
Existing deep video understanding (DVU) datasets for story videos are small, short-video-biased, and do not balance questions across story topics (character, action, location and combinations) and cognitive attributes (perception, inference), so they cannot comprehensively assess storyline understan
Approach
An LLM multi-agent pipeline (Gemini 1.5 Pro generator with a topic-balancing feedback loop from a question database, plus Gemini 1.5 Pro and Claude 3.5 Sonnet as reviewers) generates and filters questions using script, timeline and character-box information, with cross-episode questions and a video-length/content difficulty score.
Key results
- Across 10 SOTA VideoQA models on FriendsQA, average accuracy is 32.51% on single-episode and 34.72% on cross-episode questions, with VideoLLaMA2 best at 47.12% single-episode overall.
- Perception (P) questions are substantially harder than inference (I) questions for every model, and character-related perception questions show lower accuracy.
- Manual verification of FriendsQA-S1 vs FriendsQA-M shows an average accuracy difference of only 0.15% (single) and 0.25% (cross) across the 10 models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the manually revised FriendsQA-M dataset instead of FriendsQA-S1 | 10 SOTA VideoQA models · FriendsQA single-episode questions | average accuracy difference: 0.15% | No clear effect |
| Use the manually revised FriendsQA-M dataset instead of FriendsQA-S1 | 10 SOTA VideoQA models · FriendsQA cross-episode questions | average accuracy difference: 0.25% | No clear effect |
| Increase question difficulty | 10 SOTA VideoQA models · FriendsQA difficulty levels | accuracy | Hurt |
| Evaluate on inference questions rather than perception questions | 10 SOTA VideoQA models · FriendsQA | accuracy | Helped |
Surprising findings
- SeViLA, the only model fine-tuned on TVQA (which includes Friends), shows no significant accuracy difference across difficulty levels, suggesting prior exposure to the show helps.
- VideoChat2 drops from 61.70% on NExT-QA to 44.05% on FriendsQA, showing a large gap between factoid and DVU performance.
Limitations
- Dataset is built from a single sitcom (Friends), limiting genre and storytelling-style diversity.
- Question generation and review depend on proprietary LLMs (Gemini 1.5 Pro, Claude 3.5 Sonnet), and manual verification covered only the first season subset.
Experimental setup
Datasets: PAINS script dataset Friends episodes MSRVTT-QA
Benchmarks: FriendsQA
Metrics: accuracy
Baselines: Chat-UniVi MA-LMM MovieChat SeViLA TimeChat Video-ChatGPT VideoChat2 VideoLLaMA2 VIOLETv2 Vid-TLDR
Builds on: NExT-QA TVQA MovieChat PAINS
Closest papers by meaning
- MovieQA: Understanding Stories in Movies Through Question-Answering (CVPR 2016 · similarity 0.90)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.87)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.86)
- Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models (ICLR 2025 · similarity 0.86)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.86)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.