Research map / Video / Video generation
T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces T2V-CompBench, a 1400-prompt benchmark across seven compositional categories for text-to-video generation, with MLLM-, detection-, and tracking-based metrics validated against human judgments, and evaluates 23 T2V models.
Problem
Text-to-video models' ability to compose multiple objects, attributes, actions, motions, and quantities is unexplored, and existing T2V benchmarks and metrics (FID, FVD, CLIPScore) do not evaluate compositionality with multi-object prompts.
Approach
A seven-category compositional T2V prompt suite generated by GPT-4 from high-frequency real-user words, paired with category-specific evaluators (Grid-LLaVA, D-LLaVA, GroundingDINO-based rules, and DOT tracking-based motion direction) chosen by correlation with human ratings.
Key results
- Proposed metrics achieve best human correlation per category, e.g., Grid-LLaVA Spearman 0.7893 on consistent attribute binding and DOT Spearman 0.5366 on motion binding.
- Across 23 T2V models, dynamic attribute binding is the hardest category; D-LLaVA scores stay low (about 0.005 to 0.069) for all models.
- Commercial PixVerse-V3 achieves the top scores on several categories (e.g., 0.8722 on action binding via Grid-LLaVA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt VideoCrafter2 into T2V-Turbo-V2 | VideoCrafter2 · Action binding | Grid-LLaVA | Helped |
| Replace AnimateDiff with MagicTime | AnimateDiff · Dynamic attribute binding | D-LLaVA: 0.0097 -> 0.0151 | Helped |
| Add LLM-guided layout planning in LVD | ModelScope · Spatial relationships | G-Dino: 0.4118 -> 0.5405 | Helped |
Surprising findings
- Video LLM Grid-LLaVA is not uniformly better than image LLM LLaVA; LLaVA is competitive in several categories.
- Video-level PLLaVA correlates poorly with humans on several categories, including negative correlation for action binding.
Limitations
- Prompts are limited to high-frequency thing-category nouns and 200 verbs, so coverage of rare concepts is limited.
- Automatic metrics rely on detectors and MLLMs whose own errors affect scores; correlations with human judgment are moderate for some categories (e.g., D-LLaVA tau 0.43, motion DOT 0.45).
Experimental setup
Datasets: VidProM
Benchmarks: T2V-CompBench
Metrics: Kendall's tau Spearman's rho compositional category scores
Baselines: CLIPScore BLIP-CLIP BLIP-BLEU BLIP-VQA ViCLIP PLLaVA LLaVA VPEval-S M-GDino
Builds on: T2I-CompBench VBench EvalCrafter VPEval VideoDirectorGPT
Closest papers by meaning
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.91)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.90)
- Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (NeurIPS 2024 · similarity 0.90)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.89)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.89)
- Compositional 3D-aware Video Generation with LLM Director (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.