Research map / Video / Video generation
FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, Lu Hou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CLIP on MSR-VTT video-text retrieval (CLIPScore-ft) | CLIPScore · FETV video-text alignment, all categories | Kendall tau_c: 0.177 -> 0.224 | Improved |
| replace CLIP with BLIP (BLIPScore) | CLIPScore · FETV video-text alignment, all categories | Kendall tau_c: 0.177 -> 0.235 | Improved |
| replace CLIP with video-pretrained UMT (UMTScore) | CLIPScore · FETV video-text alignment, all categories | Kendall tau_c: 0.177 -> 0.309 | Improved |
| replace CLIP with UMT (UMTScore) | BLIPScore · FETV video-text alignment, all categories | Kendall tau_c: 0.235 -> 0.309 | Improved |
| use Video LLM Otter with yes/no VQA instead of CLIP similarity | CLIPScore · FETV video-text alignment, all categories | Kendall tau_c: 0.177 -> 0.081 | Hurt |
| UMTScore Quantity vs Color categories | UMTScore · FETV video-text alignment, all categories | Kendall tau_c: 0.394 vs 0.304 | Mixed |
| replace I3D encoder in FVD with UMT vision encoder (FVD-UMT) | FVD · FETV video-text alignment, all categories | ranking consistency with human quality | Improved |
| increase number of generated videos for FID/FVD (ablation) | FID/FVD · FETV video-text alignment, all categories | FID/FVD | Mixed |
| zero-shot ZeroScope trained from ModelScopeT2V vs original ModelScopeT2V | ZeroScope · FETV video-text alignment, all categories | human alignment | Hurt |
| prompt complexity simple to complex (ablation) | T2V models · FETV video-text alignment, all categories | human alignment | No clear effect |
| use fine-grained vs overall alignment rating | manual evaluation · FETV video-text alignment, all categories | human alignment | No clear effect |
Closest papers by meaning
- Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (NeurIPS 2024 · similarity 0.91)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
- Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability, Reproducibility, and Practicality (NeurIPS 2024 · similarity 0.89)
- Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.89)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.88)
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
