Research map / Video / Video generation
LVD-2M: A Long-take Video Dataset with Temporally Dense Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LVD-2M, a 2M-clip long-take video dataset with temporally dense captions, via scene-cut and optical-flow filtering, MLLM quality filtering, and hierarchical image-grid captioning, and shows fine-tuning on it improves dynamic motion in video generators.
Problem
Training long video generation models is hampered by the lack of high-quality long-take videos with rich, temporally dense text annotations; existing web video datasets contain scene cuts, static clips, and short sparse captions.
Approach
A data curation pipeline that filters long-take, high-motion videos using tuned PySceneDetect cut detection, RAFT optical-flow thresholds, and PLLaVA-based semantic filtering, then captions 30-second clips via 2x3 image grids with LLaVA-1.6-34B and merges them with Claude3-Haiku into temporally dense captions.
Key results
- Human raters judged 77.5% of LVD-2M clips long-take versus 47.5% (InternVid), 50.0% (Panda-70M), and 55.0% (HD-VG).
- Fine-tuning an I2V diffusion model on LVD-2M won 60% of user-study comparisons on dynamic degree versus 6% for the pretrained model.
- A T2V diffusion model extended to 65 frames on LVD-2M beat WebVid-10M on 10 of 16 VBench metrics, with dynamic degree 28.06% vs 20.00%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Dynamic Degree: 20.00% -> 28.06% | Helped |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Object Class: 76.95% -> 86.93% | Helped |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Human Action: 61.40% -> 76.20% | Helped |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Background Consistency: 98.02% -> 96.92% | Hurt |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Imaging Quality: 72.63% -> 70.72% | Hurt |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Multiple Objects: 26.02% -> 22.76% | Hurt |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Aesthetic Quality: 58.02% -> 57.56% | Hurt |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Motion Smoothness: 97.87% -> 98.43% | Helped |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Temporal Flickering: 98.00% -> 97.44% | Hurt |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Subject Consistency: 95.81% -> 96.12% | Helped |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Color: 75.51% -> 79.32% | Helped |
| Fine-tune the T2V diffusion model on LVD-2M instead of WebVid-10M | 1.75B T2V diffusion model · VBench | Overall Consistency: 21.43% -> 21.29% | Hurt |
Surprising findings
- Fine-tuning on LVD-2M did not improve text-video alignment for the I2V model, which the authors attribute to CLIP text encoder limits on long captions.
- Captions in LVD-2M average 88.7 words, far longer than the 7.6-word VBench prompts, yet the model still achieved better overall VBench results.
Limitations
- Dataset at 2M clips is smaller than other large video datasets.
- Human raters found remaining failures are slight jump cuts that automatic scene detection and MLLM filters miss.
- Fine-tuning experiments use only older base models with CLIP text encoders, so results on stronger T2V models are not validated.
Experimental setup
Datasets: HD-VG-130M InternVid Panda-70M WebVid-10M VBench
Benchmarks: VBench
Metrics: VBench dimensions (subject consistency, dynamic degree, obje human win rate on dynamic degree and text alignment long-take ratio from human raters average optical flow magnitude caption word count
Baselines: WebVid-10M fine-tuning Panda-70M captioner VBench evaluation of pretrained models
Builds on: PySceneDetect RAFT PLLaVA LLaVA-1.6 Claude3-Haiku IG-VLM
Closest papers by meaning
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.91)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.90)
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.90)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.89)
- LongDiff: Training-Free Long Video Generation in One Go (CVPR 2025 · similarity 0.88)
- Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.