Research map / Vision-language and multimodal / Multimodal LLMs
ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ShotBench, an expert-annotated benchmark for VLM understanding of cinematic shot language, the ShotQA training set (~70k QA pairs), and ShotVL (Qwen2.5-VL-3B tuned with SFT then GRPO), which outperforms GPT-4o and Qwen2.5-VL-72B on ShotBench.
Problem
Current VLMs' comprehension of cinematography (shot size, framing, camera angle, lens, lighting, composition, camera movement) is underexplored and lacks rigorous evaluation, limiting fine-grained visual understanding and AI-assisted video generation.
Approach
A dedicated expert-annotated multiple-choice benchmark across eight cinematography dimensions drawn from Oscar-nominated films, plus a large-scale training set and a 3B VLM trained with SFT followed by GRPO using a rule-based correctness reward.
Key results
- ShotVL (3B) reaches 65.1% average accuracy on ShotBench, versus 59.3% for GPT-4o and 59.1% for Qwen2.5-VL-72B-Instruct.
- ShotVL improves on its Qwen2.5-VL-3B base (46.1%) by 19 points on average.
- In ablations on images, SFT followed by GRPO gives the best average (55.5%) among the five training strategies compared.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with SFT and GRPO on ShotQA | Qwen2.5-VL-3B-Instruct · ShotBench | accuracy (Avg): 46.1 -> 65.1 | Helped |
| Add SFT (ablation) | Qwen2.5-VL-3B-Instruct · ShotBench | accuracy (Avg): 45.3 -> 54.3 | Helped |
| Add CoT to SFT (ablation) | Qwen2.5-VL-3B-Instruct · ShotBench | accuracy (Avg): 54.3 -> 45.4 | Hurt |
| Add GRPO without SFT (ablation) | Qwen2.5-VL-3B-Instruct · ShotBench | accuracy (Avg): 45.3 -> 53.5 | Helped |
| Add GRPO after SFT and CoT-SFT (ablation) | Qwen2.5-VL-3B-Instruct · ShotBench | accuracy (Avg): 45.4 -> 53.8 | Helped |
| Add GRPO after SFT, without CoT (ablation) | Qwen2.5-VL-3B-Instruct · ShotBench | accuracy (Avg): 54.3 -> 55.5 | Helped |
| Add CoT-SFT before GRPO instead of using SFT→GRPO (ablation) | Qwen2.5-VL-3B-Instruct · ShotBench | accuracy (Avg): 55.5 -> 53.8 | Hurt |
Surprising findings
- Chain-of-thought SFT with Gemini-generated reasoning yields only small gains over plain SFT, which the authors attribute to noisy reasoning chains.
- Camera movement is near random chance (~25%) for most evaluated models.
Limitations
- Only Qwen2.5-VL-3B was used as the base model, so scaling to larger bases is untested.
- Class imbalance, e.g., rare camera movements like dolly zoom, and reliance on real movie data.
- Annotation of video is labor-intensive, limiting scalability.
Experimental setup
Datasets: ShotQA ShotBench
Benchmarks: ShotBench
Metrics: accuracy
Baselines: GPT-4o Qwen2.5-VL-72B-Instruct Qwen2.5-VL-3B-Instruct Gemini-2.5-flash InternVL3-78B LLaVA-Video-7B
Builds on: GRPO DeepSeek-R1 Qwen2.5-VL VLMEvalKit ms-swift
Closest papers by meaning
- CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (NeurIPS 2025 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.