Research map / Video / Video generation
OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenHumanVid, a 52.3M-clip (13.2M filtered) human-centric video dataset with short/long/structured captions, skeleton and speech-audio conditions, and pretrains a LoRA-extended CogVideoX DiT on it.
Problem
General video generation models and existing video datasets lack high-quality, human-centric data with fine-grained captions and motion conditions (skeleton, speech audio), limiting appearance consistency, motion alignment, and text support for human videos.
Approach
Build a large human-centric video dataset from films, TV series and documentaries with multi-format captions (generated by MiniCPM and CogVLM, voted with BLIP2, reformatted by Llama 3.1), DWPose skeletons and SyncNet-filtered speech audio, then further pretrain CogVideoX via LoRA using human-specific quality filters and a higher frame rate.
Key results
- Extended CogVideoX pretrained on 6.05K hours of OpenHumanVid improves Human Consistency from 0.7108 to 0.7418 and Human Semantics from 0.9405 to 0.9678 over the baseline.
- Higher sampling rate (24 FPS vs 8 FPS) raises face consistency from 0.7209 to 0.7390 on the 1.05K-hour ablation.
- Human appearance filter raises human consistency from 0.7976 to 0.8674 versus no filter in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Further pretrain CogVideoX on OpenHumanVid with the proposed training strategies | CogVideoX · Human video generation test set | Face Consistency: 0.7108 -> 0.7418 | Helped |
| Use the human appearance text-video alignment filter (ablation) | Extended CogVideoX · Human video generation test set | Face Consistency: 0.7976 -> 0.8674 | Helped |
| Use the human motion text-video alignment filter (ablation) | Extended CogVideoX · Human video generation test set | Body Semantics: 0.4437 -> 0.4866 | Helped |
| Use the face motion text-video alignment filter (ablation) | Extended CogVideoX · Human video generation test set | Face Semantics: 0.3148 -> 0.3220 | Helped |
| Increase training video sampling rate from FPS = 8 to FPS = 24 (ablation) | Extended CogVideoX · Human video generation test set | Face Consistency: 0.7209 -> 0.7390 | Helped |
Limitations
- Evaluation relies on automatic metrics (BLIP-2 scores, InsightFace, DINO) and a small test set of 240 samples plus 100 human-dominant samples.
- Source data is restricted to films, TV series and documentaries, which may limit diversity relative to internet video.
Experimental setup
Datasets: OpenHumanVid
Benchmarks: VBench
Metrics: face consistency body consistency body semantics face semantics I2V consistency aesthetic quality imaging quality motion smoothness
Baselines: CogVideoX
Builds on: CogVideoX LoRA VBench DWPose BLIP-2
Closest papers by meaning
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.90)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.88)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.88)
- HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation (CVPR 2025 · similarity 0.88)
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.