Research map / Video / Video generation
Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Stable Cinemetrics (SCINE), a hierarchical 76-node cinematic-control taxonomy with expert-annotated evaluation of 13 text-to-video models and a fine-tuned Qwen2.5-VL-7B evaluator.
Problem
Existing video generation benchmarks do not measure the fine-grained cinematic controls (setup, events, lighting, camera) that professionals need, so it is unclear whether current text-to-video models are ready for professional use.
Approach
A four-pillar hierarchical taxonomy (Setup, Event, Lighting, Camera) of 76 control nodes grounded in filmmaking practice, used to generate prompts (scripts upsampled into visuals) and per-node yes/no-style rating questions for expert annotation, plus a VLM trained on expert ratings to automate evaluation.
Key results
- The fine-tuned Qwen2.5-VL-7B evaluator reaches 72.36% pairwise accuracy against annotator averages, about a 20% absolute gain over the zero-shot 7B baseline.
- Across 13 T2V models with 84 film-professional annotators, Minimax and Wan 14B are the top performers, while Events and Camera are the weakest pillars.
- Annotators agreed on model preferences: ICC 80.4% at model-pair level and statistically significant preferences in 37 of 45 model pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen-2.5-VL-7B on expert annotations | Qwen-2.5-VL-7B evaluator · SCINE video pair preferences | pairwise preference accuracy: ∼20% | Helped |
| Increase Qwen2.5-VL-Instruct model size from 7B to 32B or 72B | Zero-shot VLM evaluation · SCINE video pair preferences | alignment with human judgment | No clear effect |
| Display two generated videos side by side to annotators (ablation) | Human annotation interface · SCINE human evaluation | annotator calibration | Helped |
Surprising findings
- Increasing zero-shot VLM size from 7B to 72B does not yield significant improvements in alignment with human judgment.
- Models perform worse on advanced prompts for camera and lighting, with the largest drops in Lighting Source.
Limitations
- Expert pool is limited by the authors' collaborator network and may not capture regional or cultural filmmaking terminology.
- Some fine-grained values (e.g., exact color temperature in Kelvin, ISO) were abstracted because annotators could not perceive them consistently.
- Zero-shot VLM experiments were bounded by compute and data resources.
Experimental setup
Datasets: SCINE Scripts SCINE Visuals expert human annotations (248,536 pairwise annotations)
Benchmarks: SCINE Scripts SCINE Visuals
Metrics: 1-5 expert ratings intra-class correlation pairwise preference accuracy Wilcoxon signed-rank test
Baselines: VBench-2.0 MovieGenBench Qwen2.5-VL-7B zero-shot Qwen2.5-VL-32B zero-shot Qwen2.5-VL-72B zero-shot
Builds on: VBench Bradley-Terry model
Closest papers by meaning
- CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (NeurIPS 2025 · similarity 0.90)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.88)
- CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generation (NeurIPS 2026 · similarity 0.87)
- Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.87)
- VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 · similarity 0.87)
- CCEdit: Creative and Controllable Video Editing via Diffusion Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.