Research map / Video / Video-language understanding
ALLVB: All-in-One Long Video Understanding Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ALLVB, a 1,376-movie long video QA benchmark (252k multiple-choice QAs across 9 task types and 91 templates) generated automatically from movie scripts with GPT-4o and human-reviewed, then evaluates 12 MLLMs.
Problem
Existing video understanding benchmarks are mostly short, and long-video benchmarks rely on costly manual annotation with limited scale and inconsistent question design, so they cannot thoroughly evaluate MLLMs' long-sequence modeling.
Approach
Converts nine established video understanding tasks into 91 question templates and uses GPT-4o to generate multiple-choice QAs from movie scripts segmented hierarchically into plots and sub-plots, with script-based automated checks and a three-stage manual review.
Key results
- Claude 3.5 Sonnet achieves the highest average accuracy of 75.2% on ALLVB, with GPT-4o second at 68.0%.
- Open-source 7B video MLLMs score between 23.1% and 47.1% average accuracy, with TimeChat at 47.1% and LLaMA-VID at 46.5%.
- Increasing input frames from 16 to 64 improves GPT-4o average accuracy only from 68.0% to 68.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase input frames from 16 to 64 (ablation) | MovieChat · ALLVB test | average accuracy (%): 24.4 -> 26.0 (+1.6) | Helped |
| increase input frames from 16 to 64 (ablation) | LLaMA-VID · ALLVB test | average accuracy (%): 46.5 -> 46.7 (+0.2) | Helped |
| increase input frames from 16 to 64 (ablation) | TimeChat · ALLVB test | average accuracy (%): 47.1 -> 47.7 (+0.6) | Helped |
| increase input frames from 16 to 64 (ablation) | GPT-4o · ALLVB test | average accuracy (%): 68.0 -> 68.5 (+0.5) | Helped |
| increase video length from 80-90 min to 130-140 min (ablation) | Otter-V · ALLVB videos by duration | accuracy (%): 24.7 -> 22.5 | Hurt |
| increase video length from 80-90 min to 130-140 min (ablation) | VideoChat2 · ALLVB videos by duration | accuracy (%): 38.5 -> 37.7 | Hurt |
| increase video length from 80-90 min to 130-140 min (ablation) | LLaMA-VID · ALLVB videos by duration | accuracy (%): 47.8 -> 45.9 | Hurt |
| increase video length from 80-90 min to 105-115 min (ablation) | GPT-4o · ALLVB videos by duration | accuracy (%): 70.3 -> 66.9 | Hurt |
| increase video length from 105-115 min to 130-140 min (ablation) | GPT-4o · ALLVB videos by duration | accuracy (%): 66.9 -> 66.9 (-0.0) | No clear effect |
| use open-source video MLLMs rather than image MLLMs under the reported 7B input conditions | video MLLMs · ALLVB test | average accuracy (%) | Mixed |
Surprising findings
- Open-source video MLLMs show no significant advantage over image MLLMs of the same 7B size under the same input conditions.
- Going from 16 to 64 input frames yields only marginal gains (0.2 to 1.6 points), suggesting models do not fully exploit extra long-video information.
Limitations
- All videos are sourced from movies, lacking sports, documentary, or instructional content.
- Evaluation feeds only 16 uniformly sampled frames plus subtitles, limiting access to video information.
- Question content is relatively simple compared with human capabilities; GPT-4o annotations may raise fairness concerns (script-based generation).
Experimental setup
Datasets: ALLVB
Benchmarks: ALLVB
Metrics: accuracy
Baselines: Otter-I LLaVA-1.6 GPT4-Turbo Otter-V mPlug-Owl-V MovieChat VideoChat VideoChat2 LLaMA-VID TimeChat
Builds on: MLLaVA-style video MLLMs Gemini 1.5 Pro needle-in-a-haystack test
Closest papers by meaning
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.92)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- MLVU: Benchmarking Multi-task Long Video Understanding (CVPR 2025 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.89)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.