Research map / Video / Video-language understanding
HourVideo: 1-Hour Video-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HourVideo, a 12,976-question five-way MCQ benchmark over 500 hour-long Ego4D videos spanning 18 sub-tasks, and evaluates blind LLMs, Socratic captioning pipelines, and native multimodal models, finding Gemini 1.5 Pro at 37.3% vs. 85% human.
Problem
Existing video-language benchmarks focus on short clips (seconds to minutes) and can be answered from a single frame or prior knowledge, so they do not test long-term comprehension of hour-long videos.
Approach
A task suite of 4 task families and 18 sub-tasks (summarization, perception, visual reasoning, navigation) with question prototypes, built via an LLM-plus-human pipeline on Ego4D narrations, with blind-LLM filtering to remove questions answerable without video.
Key results
- Gemini 1.5 Pro achieves 37.3% average accuracy on HourVideo versus 85.0% for human experts and 20% random chance.
- GPT-4 blind LLM scores 19.6%, and the GPT-4 Socratic pipeline scores 25.7%, close to chance.
- Tarsier-7B (16 frames) reaches 26.7% average accuracy, comparable to Socratic LLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini 1.5 Pro for native multimodal video evaluation instead of GPT-4 Socratic captioning | HourVideo · HourVideo | accuracy: 25.7 -> 37.3 | Helped |
| Use GPT-4 Socratic captioning instead of blind GPT-4 | GPT-4 · HourVideo | accuracy: 19.6 -> 25.7 | Helped |
| Evaluate each MCQ independently rather than grouping questions by task (ablation) | Gemini 1.5 Pro evaluation on HourVideo · HourVideo subset of 570 MCQs | performance: 38.9\% -> 36.8\% | Hurt |
| Evaluate each MCQ independently rather than grouping questions by task (ablation) | Gemini 1.5 Pro evaluation on HourVideo · HourVideo subset of 570 MCQs | evaluation cost: \$846 -> \$2621 | Hurt |
Surprising findings
- Gemini 1.5 Pro refuses 16.45% of questions, far more than GPT-4 (under 0.4%).
- Blind GPT-4 scores near chance (19.6%), indicating residual shortcuts or prior knowledge are limited but nonzero.
Limitations
- Some inconsistencies may remain in the MCQs despite human refinement.
- Only egocentric Ego4D videos; lacks sports and YouTube sources and audio modality.
- Evaluation relies on proprietary models, and Gemini refusals affect coverage.
Experimental setup
Datasets: Ego4D
Benchmarks: HourVideo
Metrics: accuracy
Baselines: GPT-4 blind LLM Socratic GPT-4 Socratic LLaVA-NeXT-34B-DPO Gemini 1.5 Pro Tarsier-7B
Builds on: Socratic models Ego4D Chain-of-thought prompting
Closest papers by meaning
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.93)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.