Research map / Video / Video generation
VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds VideoUFO, a ~1.09M-clip text-to-video dataset whose topics are mined from real user prompts (VidProM) and sourced from CC-licensed YouTube videos, plus a BenchUFO benchmark and an MVDiT model trained on it that improves worst-topic scores.
Problem
Text-to-video models often fail on topics real users want to generate because their training data does not cover those user-focused topics; existing video datasets are open-domain and many have restrictive or retracted licenses.
Approach
Derive user-focused topics by clustering real user prompts from VidProM, retrieve CC-licensed YouTube videos for each topic via the official API, split and caption clips, verify topic presence with GPT-4o-mini on detailed captions, and benchmark models on worst/best topics using caption-prompt similarity.
Key results
- Current 16 text-to-video models show a 0.233 to 0.314 score gap between top-10 and low-10 user-focused topics.
- MVDiT trained on VideoUFO achieves the highest low-10 score, a +4.2% improvement over the current state of the art, while maintaining top-10 performance.
- Only 0.29% (1,675) of YouTube IDs overlap with existing datasets; detailed captions with GPT-4o-mini verification reduce cost by roughly 5,400x versus direct GPT-4o while keeping about 80% overlap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train MVDiT solely on VideoUFO | MVDiT · BenchUFO low-10 topics | similarity: +4.2% | Helped |
| Replace VideoUFO with OpenVid for training (ablation) | MVDiT · BenchUFO low-10 topics | similarity: 0.401 vs. 0.400 | Hurt |
| Replace VideoUFO with VidGen for training (ablation) | MVDiT · BenchUFO low-10 topics | similarity | Hurt |
| Use detailed captions instead of videos for verification (ablation) | Clip verification · 1,000 clips | overlap with Direct GPT-4o: 0.770; 0.806 | Hurt |
| Use brief captions instead of videos for verification (ablation) | Clip verification · 1,000 clips | overlap with Direct GPT-4o: 0.621; 0.657 | Hurt |
| Use GPT-4o-mini instead of GPT-4o on detailed captions (ablation) | Clip verification · 1,000 clips | overlap with Direct GPT-4o: 0.770 -> 0.806 | Helped |
| Increase the preset K-means cluster count beyond approximately 2,000 (ablation) | User-focused topic extraction · VidProM prompts | number of resulting topics | No clear effect |
| Use DBSCAN or HDBSCAN to determine the cluster count (ablation) | User-focused topic extraction · VidProM prompts | topic extraction quality | Hurt |
Surprising findings
- OpenVid-1M-trained MVDiT reaches low-10 scores similar to prior state of the art (0.401 vs 0.400), and VidGen-1M training performs even worse.
- GPT-4o-mini verification slightly outperforms GPT-4o on the clip-verification overlap test.
Limitations
- Training validation uses an academic baseline video generator requiring 32 A100 GPUs, whose output quality remains limited.
- Topic analysis for other datasets relies on their existing captions, which may be uninformative and affect topic counts.
- Benchmark uses caption-prompt similarity via a VLM, and abstract nouns were excluded due to mismatch issues.
Experimental setup
Datasets: VidProM VideoUFO OpenVid-1M VidGen-1M VBench
Benchmarks: BenchUFO
Metrics: caption-prompt cosine similarity average similarity on low-10 topics average similarity on top-10 topics VBench quality scores
Baselines: OpenVid-1M VidGen-1M Sora CogVideoX HunyuanVideo LTX-Video Mochi-1
Builds on: VidProM Panda-70M Open-Sora-Plan VBench MVDiT
Closest papers by meaning
- VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmark (AAAI 2025 · similarity 0.88)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.88)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.88)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.88)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.