Research map / Video / Video-language understanding
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LongVideoBench, a human-annotated multiple-choice benchmark of 6,678 referring-reasoning questions over 3,763 subtitled videos up to an hour long, and evaluates proprietary and open-source LMMs across frame counts.
Problem
Existing video QA benchmarks show single-frame bias and do not reward processing more frames, and few benchmarks test long-context interleaved video-subtitle understanding.
Approach
A "referring reasoning" task where each question first points to a referred moment (scene, event, object, or subtitle) in a long video and then asks a question about that context, with 17 categories at perception and relation levels, so that accuracy improves only when models process more frames.
Key results
- GPT-4o reaches 66.7% on the LongVideoBench test set, with validation accuracy rising from 41.7% at 1 frame to 66.7% at 256 frames.
- Open-source models such as Idefics2 and Mantis-Idefics2 degrade at 64 frames, while proprietary models keep improving with more frames.
- Test accuracy of the best open-source model PLLaVA-34B is 53.5%, and all models score lowest on the Sequence of Scenes category and the 900-3600s duration group.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase GPT-4o max_frames from 1 to 256 (ablation) | GPT-4o · LongVideoBench validation, all durations | accuracy: 41.7 -> 66.7 | Helped |
| increase Gemini-1.5-Pro max_frames from 1 to 256 (ablation) | Gemini-1.5-Pro · LongVideoBench validation, all durations | accuracy: 38.6 -> 64.0 | Helped |
| increase GPT-4-Turbo max_frames from 1 to 256 (ablation) | GPT-4-Turbo · LongVideoBench validation, all durations | accuracy: 43.2 -> 59.0 | Helped |
| increase Gemini-1.5-Flash max_frames from 1 to 256 (ablation) | Gemini-1.5-Flash · LongVideoBench validation, all durations | accuracy: 38.1 -> 61.6 | Helped |
| increase Idefics2 max_frames from 16 to 64 (ablation) | Idefics2 · LongVideoBench validation, all durations | accuracy: 49.7 -> 30.9 | Hurt |
| increase Mantis-Idefics2 max_frames from 16 to 64 (ablation) | Mantis-Idefics2 · LongVideoBench validation, all durations | accuracy: 47.0 -> 30.2 | Hurt |
| remove video frames and provide subtitles only (ablation) | GPT-4o · LongVideoBench validation | accuracy: 66.7 -> 44.6 | Hurt |
| remove text subtitles and provide video frames only (ablation) | GPT-4o · LongVideoBench validation | accuracy: 66.7 -> 60.6 | Hurt |
| scale the language-model backbone from 7B to 13B (ablation) | PLLaVA · LongVideoBench test | accuracy: 39.2 -> 45.1 | Helped |
| scale the language-model backbone from 7B to 34B (ablation) | PLLaVA · LongVideoBench test | accuracy: 39.2 -> 53.5 | Helped |
Surprising findings
- Open-source video LMMs do not outperform same-architecture image LMMs; LLaVA-Next-Video-M7B trails LLaVA-Next-Mistral.
- Idefics2 and Mantis-Idefics2 lose accuracy when given 64 frames, before reaching their context limits.
Limitations
- Scope limited to vision and language; audio is not included.
- Videos longer than one hour are not evaluated.
- Evaluation is zero-shot only, with no error bars or training experiments.
Experimental setup
Datasets: LongVideoBench
Benchmarks: LongVideoBench
Metrics: accuracy
Baselines: GPT-4o Gemini-1.5-Pro GPT-4-Turbo Gemini-1.5-Flash Idefics2 Phi-3-Vision-Instruct Mantis-Idefics2 Mantis-BakLLaVA PLLaVA-34B LLaVA-Next-Video-34B
Builds on: needle in a haystack
Closest papers by meaning
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.93)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.93)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.92)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.92)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.