Research map / Video / Video generation
MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MiraData, a 16K-hour long-video dataset with structured captions from GPT-4V, plus the MiraBench evaluation suite and a DiT-based MiraDiT model to show gains in motion strength and 3D consistency.
Problem
Existing public text-video datasets have short clips, low motion intensity, and brief captions, which are insufficient for training Sora-like long video generation models.
Approach
Curate long, high-motion, high-quality clips from manually selected YouTube channels and stock sites, split and stitch them with a multi-model consensus, filter by color/aesthetics/motion/NSFW, and annotate with GPT-4V structured captions (short, dense, main object, background, camera, style).
Key results
- MiraDiT trained on MiraData reaches dynamic degree 15.46 and tracking strength 49.47 vs 7.12 and 22.36 when trained on WebVid-10M.
- MiraData averages 72.1-second videos and 318-word captions, versus 12-30 word captions in prior datasets.
- Structured captions raise tracking strength to 68.85 vs 27.03 for short captions in MiraDiT ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train MiraDiT on MiraData instead of WebVid-10M | MiraDiT · MiraBench | Dynamic Degree: 7.12 -> 15.46 | Helped |
| Train MiraDiT on MiraData instead of WebVid-10M | MiraDiT · MiraBench | Tracking Strength: 22.36 -> 49.47 | Helped |
| Train MiraDiT on MiraData instead of WebVid-10M | MiraDiT · MiraBench | DINO Temporal Consistency: 20.24 -> 43.78 | Helped |
| Train MiraDiT on MiraData instead of WebVid-10M | MiraDiT · MiraBench | Mean Absolute Error: 91.48 -> 85.27 | Helped |
| Train MiraDiT on MiraData instead of WebVid-10M | MiraDiT · MiraBench | Aesthetic Quality: 43.11 -> 49.90 | Helped |
| Train MiraDiT on MiraData instead of WebVid-10M | MiraDiT · MiraBench | Main Object Alignment: 14.32 -> 14.67 | Helped |
| Train MiraDiT on MiraData instead of WebVid-10M | MiraDiT · MiraBench | Overall Alignment: 15.31 -> 16.66 | Helped |
| Use dense captions instead of short captions (ablation) | MiraDiT · MiraBench | Dynamic Degree: 9.45 -> 17.39 | Helped |
| Use structural captions instead of short captions (ablation) | MiraDiT · MiraBench | Tracking Strength: 27.03 -> 68.85 | Helped |
| Use structural captions instead of short captions (ablation) | MiraDiT · MiraBench | DINO Temporal Consistency: 24.39 -> 60.83 | Helped |
| Use structural captions instead of short captions (ablation) | MiraDiT · MiraBench | Overall Alignment: 7.73 -> 15.36 | Helped |
| Use structural captions instead of short captions (ablation) | MiraDiT · MiraBench | Aesthetic Quality: 4.84 -> 4.99 | Mixed |
Surprising findings
- Longer and more detailed captions do not necessarily improve visual quality; short captions gave the highest aesthetic score (4.84 vs 4.99 structural), though the paper's aesthetic metric scale is unclear.
Limitations
- Inherent biases and potential annotation errors from GPT-4V captions.
- MiraBench metrics may be inaccurate in uncommon scenarios such as jitter or overexposure.
- Insufficient coverage of video domains.
Experimental setup
Datasets: MiraData WebVid-10M
Benchmarks: MiraBench
Metrics: dynamic degree tracking strength DINO temporal consistency CLIP temporal consistency temporal motion smoothness MAE RMSE aesthetic quality imaging quality camera alignment
Baselines: OpenSora VideoCrafter2 MiraDiT trained on WebVid-10M
Builds on: Diffusion Transformer PixArt-alpha SVD VBench Panda-70M
Closest papers by meaning
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.87)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.87)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.87)
- Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content (CVPR 2025 · similarity 0.87)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.86)
- ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.