Research map / Video / Video-language understanding
VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VidComposition, a human-annotated benchmark of 982 compiled videos with 1,706 multiple-choice questions on video composition (cinematography, character, narrative, scene, making), and evaluates 33 MLLMs, finding a large gap to humans.
Problem
Existing video benchmarks for MLLMs focus on abstract or natural video comprehension and do not assess fine-grained understanding of video compositions (camera work, shot size, narrative structure, editing) in compiled videos such as films, TV series, and vlogs.
Approach
A cinematic-level benchmark built from audio-stripped compiled commentary videos with human-written multiple-choice questions across 15 sub-tasks, plus diagnostic analyses of frame count, encoder resolution, LLM size, and SFT data volume on performance.
Key results
- Human accuracy on VidComposition is 86.26%, versus 63.31% for the best model, LLaVA-OneVision-72B.
- Open-source InternVL2 variants outperform API models such as GPT-4o (52.93) and Gemini-1.5-Flash (52.40), with the mean across 33 MLLMs at 43.44%.
- Random-choice baseline is 25.33%; models exceed it but remain far from human level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase input frames from 4 to 128 (ablation) | LongVA · VidComposition | overall accuracy: 40.74; 43.73; 43.32; 41.62; 39.98; 39.39 | No clear effect |
| Increase visual encoder resolution from 224 to 336 (ablation) | MLLMs with 7B language decoders and 8 input frames · VidComposition | overall accuracy: 34.94; 38.04+3.1 | Helped |
| Increase visual encoder resolution from 336 to 384 (ablation) | MLLMs with 7B language decoders and 8 input frames · VidComposition | overall accuracy: 38.04+3.1; 43.7+5.66 | Helped |
| Increase Qwen2-VL decoder size from 2B to 7B (ablation) | Qwen2-VL · VidComposition | overall accuracy: 36.17; 49.3+13.13 | Helped |
| Increase Qwen2-VL decoder size from 7B to 72B (ablation) | Qwen2-VL · VidComposition | overall accuracy: 49.3; 58.68+9.38 | Helped |
| Increase Video-LLaMA2 decoder size from 7B to 72B (ablation) | Video-LLaMA2 · VidComposition | overall accuracy: 34.35; 58.62+24.27 | Helped |
| Increase InternVL2 decoder size from 34B to 70B (ablation) | InternVL2 · VidComposition | overall accuracy: 60.73+14.31; 58.73−2.0 | Hurt |
| Increase SFT data volume from 0.65M to 2M (ablation) | Chat-UniVi / VideoChat2 · VidComposition | overall accuracy: 26.73; 47.01+20.28 | Helped |
| Increase SFT data volume from 1.27M to 1.32M (ablation) | Chat-UniVi-v1.5 / LongVA · VidComposition | overall accuracy: 36.11; 43.73+7.62 | Helped |
| Increase SFT data volume from 2.94M to 8.32M (ablation) | Kangaroo / MiniCPM-V · VidComposition | overall accuracy: 37.1; 42.5+5.4 | Helped |
| Increase SFT data volume from 1.21M to 3.35M (ablation) | VILA-1.5 / Video-LLaMA2.1-AV · VidComposition | overall accuracy: 40.04; 40.09+0.05 | No clear effect |
| Use Video-LLaMA2.1 instead of Video-LLaMA2.1-AV with 3.35M SFT data (ablation) | Video-LLaMA2.1-AV · VidComposition | overall accuracy: 40.09+0.05; 47.3+7.22 | Mixed |
Surprising findings
- Increasing the number of input frames does not consistently improve accuracy; performance is stable or fluctuates randomly.
- Models perform worst on cinematography tasks such as camera movement and shot size perception, despite strong results on action and costume perception.
Limitations
- Evaluation is limited to multiple-choice questions with four options, which may not capture open-ended composition understanding.
- Audio is removed from videos, so speech cues cannot be used, which departs from realistic video understanding.
Experimental setup
Datasets: VidComposition
Benchmarks: VidComposition
Metrics: accuracy
Baselines: GPT-4o GPT-4o mini GPT-4-turbo Gemini-1.5-Pro Gemini-1.5-Flash LLaVA-OneVision-72B InternVL2 Qwen2-VL Video-LLaMA2 VILA-1.5 LongVA Random
Builds on: MVBench Video-MME TempCompass Winoground MMComposition
Closest papers by meaning
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.87)
- ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.86)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.