Research map / Video / Video generation
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenVid-1M, a ~1M-clip high-quality open-scenario text-video dataset with long LLaVA-generated captions (plus a 433K 1080p subset), and a Multi-modal Video Diffusion Transformer (MVDiT) that jointly processes visual and text tokens.
Problem
Existing large text-to-video datasets (WebVid-10M, Panda-70M) contain many low-quality, static, flickering, or watermarked clips with short captions, and recent T2V diffusion transformers under-use text tokens via simple cross-attention.
Approach
Filter millions of clips by aesthetics, temporal consistency, motion (UniMatch optical flow), and clarity (DOVER), recaption with LLaVA-v1.6-34b to get long captions, and propose MVDiT, which runs parallel visual and text token streams through multimodal self-attention, multimodal temporal attention, and an added cross-attention layer.
Key results
- On 256x256 with the same STDiT model and 50K steps, OpenVid-1M gives VQA_A 17.78 vs 13.40 (WebVid-10M) and 17.08 (Panda-50M), and the best Blip_bleu of 24.93.
- At 1024x1024, OpenVidHD-0.4M reaches VQA_A 73.46 and VQA_T 68.58, with Clip_temp_score 99.87 and warping error 0.0052, outperforming Panda-50M-HD and super-resolved baselines.
- In human evaluation over 1,117 caption pairs, long LLaVA captions were preferred 80.75% of the time over Panda's short captions (mean 4.22 vs 3.71).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train STDiT on OpenVid-1M instead of Panda-50M | STDiT · Text-to-video benchmark, 256×256 | Blip_bleu: 24.06 -> 24.93 | Helped |
| Add MHCA to MVDiT (ablation) | MVDiT-256 · MVDiT-256 | VQA_A: 13.9 -> 22.39 | Helped |
| Add α scaling parameter (ablation) | MVDiT-256 · MVDiT-256 | VQA_A: 3.16 -> 22.39 | Helped |
| Add α scaling parameter (ablation) | MVDiT-256 · MVDiT-256 | Warping_error: 0.0561 -> 0.0091 | Helped |
| Use MVDiT instead of STDiT (ablation) | Video generation model trained on Ours-0.4M at 256×256 · Text-to-video benchmark, 256×256 | VQA_A: 11.11 -> 22.39 | Helped |
| Increase training resolution from 256×256 to 512×512 (ablation) | MVDiT trained on OpenVid-1M · Text-to-video benchmark | VQA_A: 24.87 -> 66.65 | Helped |
| Add aesthetics, temporal, and motion screening (ablation) | Training data pipeline · Text-to-video benchmark, 256×256 | Blip_bleu: 23.91 -> 24.43 | Helped |
| Add clarity screening (ablation) | Training data pipeline with aesthetics, temporal, and motion screening · Text-to-video benchmark, 256×256 | VQA_A: 20.32 -> 30.26 | Helped |
| Add temporal screening (ablation) | Training data pipeline · Text-to-video benchmark, 256×256 | Clip_temp_score: 99.70 -> 99.73 | Helped |
| Use OpenVidHD-0.4M instead of WebVid and YouTube training data | Video restoration model · Video restoration | SSIM: 0.6195 -> 0.7165 | Helped |
| Use OpenVidHD-0.4M instead of WebVid-10M with 4× super-resolution | STDiT · Text-to-video benchmark, 1024×1024 | VQA_A: 69.26 -> 73.46 | Helped |
| Use generated long captions instead of Panda short captions | Video captioning · Human caption preference, 1,117 validation samples | Preference: 19.25% -> 80.75% | Helped |
Surprising findings
- Panda-50M-HD (manually selected 1080p Panda clips) performs poorly (VQA_A 13.48), suggesting resolution alone does not ensure quality.
- Removing the temporal-attention module makes the model produce unrelated images rather than coherent video.
Limitations
- Model still struggles with intricate physical dynamics and natural-scene motion, producing unrealistic videos.
- Caption quality shows some hallucinations and remains weak in modeling motion, per human evaluation.
Experimental setup
Datasets: OpenVid-1M OpenVidHD-0.4M Panda-50M ChronoMagic CelebvHQ Open-Sora-plan WebVid-10M
Benchmarks: VBench-style T2V metrics (VQA_A, VQA_T, BLIP-BLEU, SD score, CLIP temporal score, warping error) video restoration on UDM10
Metrics: VQA_A VQA_T Blip_bleu SD_score Clip_temp_score warping error PSNR SSIM LPIPS DOVER
Baselines: WebVid-10M Panda-50M Latte VideoCrafter Modelscope CogVideoX-5B OpenSoraPlan-V1.2 Upscale-A-Video
Builds on: MMDiT OpenSora STDiT PixArt-alpha SVD data pipeline
Closest papers by meaning
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.91)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.91)
- OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation (NeurIPS 2025 · similarity 0.90)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.89)
- VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.