Research map / Video / Video generation
MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MovieBench, a hierarchical (movie/scene/shot) benchmark dataset of 91 LSMDC movies with character banks, audio samples, and shot-level annotations, and evaluates keyframe, identity-customized, image-to-video, and talking-human generators on it.
Problem
Long, multi-scene movie-level video generation needs character-consistent, narrative-coherent data, but existing video-text datasets consist of short clips and lack character ID and scene-level annotations.
Approach
Builds a hierarchical movie-level dataset from LSMDC films with IMDb synopses, character banks (portraits and per-character audio), LLM-generated scene and shot annotations from interleaved multimodal inputs, and new character-ID consistency metrics (Precision_ID, Recall_ID, F1_ID) computed with DeepFace.
Key results
- Character ID F1 for GPT-4o annotations reaches 93% versus 82% for Gemini-1.5-pro and 23% precision/66% recall-level performance for MiniCPM-V (MiniCPM-V F1 34%).
- Existing character-consistent generators reach at most about 53% character ID accuracy; Magic-Me reaches F1_ID 10.17% and DreamVideo 3.74% in text-to-video settings.
- MovieBench has 91 movies averaging 45.6 minutes, with shot-level clips averaging 4.09 seconds and 69.2 hours total video.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MovieBench rather than InternVid or MiraData | Video quality evaluation · MovieBench | Aesthetic Score: 9.09; 9.70 -> 20.67 | Helped |
| Use MovieBench rather than InternVid or MiraData | Video quality evaluation · MovieBench | Inception Score: 11.68; 6.27 -> 12.34 | Helped |
| Use GPT-4o instead of MiniCPMV for plot annotation | Shot-level annotation generation · Plot annotation | Completeness: 4.12 -> 4.78 | Helped |
| Use GPT-4o instead of MiniCPMV for plot annotation | Shot-level annotation generation · Plot annotation | Hallucination: 3.14 -> 1.74 | Helped |
| Use GPT-4o instead of MiniCPMV for background annotation | Shot-level annotation generation · Background annotation | Completeness: 4.36 -> 4.81 | Helped |
| Use Gemini-1.5-pro instead of MiniCPMV for background annotation | Shot-level annotation generation · Background annotation | Hallucination: 2.52 -> 1.17 | Helped |
| Use GPT-4o instead of MiniCPMV for camera annotation | Shot-level annotation generation · Camera annotation | Completeness: 4.32 -> 4.88 | Helped |
| Use GPT-4o instead of MiniCPMV for camera annotation | Shot-level annotation generation · Camera annotation | Hallucination: 2.48 -> 1.17 | Helped |
| Use GPT-4o instead of MiniCPMV for style annotation | Shot-level annotation generation · Style annotation | Completeness: 4.60 -> 4.95 | Helped |
| Use Gemini-1.5-pro instead of MiniCPMV for style annotation | Shot-level annotation generation · Style annotation | Hallucination: 1.61 -> 1.11 | Helped |
| Use GPT-4o instead of MiniCPMV for shot-level character identification | Appearing-character annotation · MovieBench shot-level character set | F1 ScoreID: 34 -> 93 | Helped |
| Use AutoStory instead of StoryGen for text-to-keyframe/storyboard generation | Text-to-keyframe/storyboard generation · MovieBench test set | F1ID: 2.80 -> 54.34 | Helped |
Surprising findings
- Text-to-video and image-to-video models show very poor character ID consistency despite strong FID/FVD scores on the benchmark.
- Movie audio descriptions cannot be directly used as plot annotations due to lack of character consistency and narrative coherence.
Limitations
- Only 6 test movies were manually corrected, and most annotations rely on GPT-4o outputs without full human verification.
- Character ID metrics depend on DeepFace recognition, which may misidentify characters.
- Relatively small number of movies (91) limits diversity.
Experimental setup
Datasets: LSMDC MAD IMDb cast lists and synopses
Benchmarks: MovieBench
Metrics: CLIP score Inception score Aesthetic score FID FVD VBench consistency metrics Precision_ID Recall_ID F1_ID Lip-sync
Baselines: StoryGen StoryDiffusion AutoStory DreamVideo Magic-Me I2VGen-XL SVD CogVideoX Edtalk Hallo2
Builds on: LSMDC MAD MiraData
Closest papers by meaning
- EntityBench: Towards Entity-Consistent Long-Range Multi-Shot Video Generation (NeurIPS 2026 · similarity 0.91)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.90)
- VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 · similarity 0.89)
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.