Research map / Video / Video generation
Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StoryEval, a 423-prompt benchmark of 2-4 consecutive-event short stories that scores text-to-video models by event completion rate judged by VLM verifiers (GPT-4o, LLaVA-OV-Chat-72B) with unanimous voting; no model exceeds 50% average.
Problem
Existing video generation benchmarks focus on detail-level quality (aesthetics, temporal consistency, video-text alignment) and do not measure whether generated videos coherently present multiple sequential events in a story, which is needed for future long video generation.
Approach
A story-oriented metric that measures the completion rate of consecutive events in short prompts, judged by VLM verifiers with a two-step describe-then-score query and unanimous (N-over-N) voting to reduce overestimation, validated against human annotation.
Key results
- Best model Kling-1.5 reaches 40.1% average completion rate on StoryEval; no model exceeds 50%.
- GPT-4o with unanimous voting achieves Kendall's tau 0.93 and Spearman's rho 0.97 against human annotation, versus 0.75/0.86 for vanilla averaging.
- LLaVA-OV-Chat-72B verifier gives rankings closely matching GPT-4o (Spearman 0.982 on average) but tends to overestimate completion rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Kling-1.5 rather than Pika-1.5 for video generation | Text-to-video generation · StoryEval | completion rate: 19.4% -> 40.1% | Helped |
| Use Hailuo rather than Pika-1.5 for video generation | Text-to-video generation · StoryEval | completion rate: 19.4% -> 35.1% | Helped |
| Use Vchitect-2.0 rather than Pika-1.5 for video generation | Text-to-video generation · StoryEval | completion rate: 19.4% -> 21.7% | Helped |
| Evaluate with LLaVA-OV-Chat-72B instead of GPT-4o | StoryEval verifier · StoryEval | completion rate: Kling-1.5: 40.1% -> 41.7%; Hailuo: 35.1% -> 41.0%; Vchitect-2.0: 21.7% -> 31.6%; Pika-1.5: 19.4% -> 25.0% | Mixed |
| Require unanimous agreement across three GPT-4o responses instead of averaging them (ablation) | StoryEval scoring · Human alignment | Kendall's τ: 0.75 -> 0.93 | Helped |
| Require unanimous agreement across three GPT-4o responses instead of averaging them (ablation) | StoryEval scoring · Human alignment | Spearman's ρ: 0.86 -> 0.97 | Helped |
| Require unanimous agreement across three GPT-4o responses instead of 1-over-3 voting (ablation) | StoryEval scoring · Human alignment | Mean Diff.: 0.22 -> 0.05 | Helped |
| Require unanimous agreement across three GPT-4o responses instead of 2-over-3 voting (ablation) | StoryEval scoring · Human alignment | Kendall's τ: 0.71 -> 0.93 | Helped |
Surprising findings
- Open-Sora-Plan 1.3.0 has good VBench temporal consistency but the worst story completion, suggesting a trade-off between temporal consistency and story presentation.
- Even retrieval-based prompts derived from real videos, where the source video achieves 100% completion, are not fully completed by any model.
Limitations
- Relies on VLM verifiers that may overestimate completion on blurry generated videos; filtered prompts where verifier alignment was weak.
- Prompts limited to short stories of 2-4 events with 1-2 subjects, exclude facial expressions and long narratives.
- Human alignment evaluated on a subset of about 900 annotated examples with per-model prompt subsets differing from full benchmark.
Experimental setup
Datasets: YouTube videos Pexels Panda-70M
Benchmarks: StoryEval
Metrics: completion rate Spearman's rho Kendall's tau temporal consistency (VBench)
Baselines: Kling-1.5 Hailuo Pika-1.5 ModelScope EasyAnimate v4 Open-Sora-Plan 1.3.0 Open-Sora 1.2 VideoCrafter2 CogVideoX-5B Vchitect-2.0 Pyramid-Flow
Builds on: VBench EvalCrafter Panda-70M
Closest papers by meaning
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.87)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.87)
- Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models (CVPR 2024 · similarity 0.87)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.86)
- MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences (ICLR 2025 · similarity 0.86)
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.