Research map / Video / Video generation
Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Koala-36M, a 36M-clip text-to-video dataset built with an SVM-based transition detector, 200-word structured captions, a learned Video Training Suitability Score (VTSS) filter, and metric conditioning in the generator.
Problem
Existing large video-text datasets suffer from imprecise temporal splitting, short and incomplete captions, and heuristic multi-threshold quality filtering, which hurt text-video consistency for video generation training.
Approach
Combines a learned color-structure SVM transition detector with a 3σ temporal test, a structured six-aspect captioning system fine-tuned from LLaVA, a multi-branch network (Swin 3D + ConvNeXt + label fusion via WCGB) that outputs one VTSS score for filtering, and AdaLN-based injection of data metrics as generator conditions.
Key results
- A text-to-video model trained on Koala-36M reaches a VBench total score of 0.7156 versus 0.6493 for the same model trained on Panda-70M.
- With metric conditioning, Koala-36M reaches a VBench total score of 0.7460 and semantic score 0.5915.
- Captions average 202 words, compared with 13.2 words for Panda-70M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Panda-70M training data with Koala-36M | The text-to-video generation model · VBench | Total Score: 0.6493 -> 0.7156 | Helped |
| Filter the Koala data using VTSS instead of using all 48M data (ablation) | Koala-w/o TSA · VBench | Total Score: 0.7140 -> 0.7156 | Helped |
| Use VTSS filtering instead of multiple manual thresholds (ablation) | Koala-37M-manual · VBench | Total Score: 0.7073 -> 0.7156 | Helped |
| Add metric conditions to Koala-36M (ablation) | Koala-36M generation model · VBench | Quality Score: 0.4504 -> 0.5915 | Helped |
| Add metric conditions to Koala-36M (ablation) | Koala-36M generation model · VBench | Total Score: 0.7156 -> 0.7460 | Helped |
Limitations
- The dataset is insufficient for training video generation models with over 1B parameters.
- Generator experiments use small 256x256, 2-second videos trained from scratch.
Experimental setup
Datasets: Panda-70M Koala-36M
Benchmarks: VBench
Metrics: VBench quality score VBench semantic score VBench total score
Baselines: Panda-70M Koala-w/o TSA Koala-37M-manual
Builds on: Panda-70M LLaVA PySceneDetect OpenSora
Closest papers by meaning
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.89)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.88)
- UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions (NeurIPS 2025 · similarity 0.87)
- MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions (NeurIPS 2024 · similarity 0.87)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.87)
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.