Research map / Video / Video generation
HumanScore: Benchmarking Human Motions in Generated Videos
Tiange Xiang, Yusu Fang, Tian Tan, Narayan Schuetz, Scott Delp, Fei-Fei Li, Ehsan Adeli
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HumanScore, a biomechanics-informed benchmark that scores human motion quality in AI-generated videos via six metrics (anatomy, kinematics, kinetics) and evaluates 13 video generators.
Problem
Existing video generation benchmarks emphasize visual realism and semantic alignment and do not systematically measure the biomechanical plausibility of human figures and motions in generated videos.
Approach
Extract 3D human pose/mesh and OpenSim skeletons from generated videos and compute six interpretable biomechanics-grounded metrics (extra limbs, bone-length stability, joint range of motion, self-collision, kinematic extremes, motion smoothness) aggregated via frequency, severity, and persistence, rather than relying on appearance or distributional metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MeTRAbs-based fitting with temporally optimized keypoints or PromptHMR for pose estimation (ablation) | HumanScore metrics · model ranking robustness | model ranking | No clear effect |
| vary tolerance scales from strict to permissive (ablation) | HumanScore metrics · model ranking robustness | model ranking | No clear effect |
| grid search over aggregation weights (alpha, beta, gamma) (ablation) | HumanScore aggregation · Wan 2.2, CogVideoX-5B, Sora-2, Veo 3.1 fast and real videos | model ranking | No clear effect |
| add systematic prompt refinement (studio, full-body, static camera, single person) | video generators · HumanScore motion realism | motion realism | No clear effect |
| compare HumanScore model ranking with human preference win ratios | HumanScore · human preference study (~1200 responses) | Spearman correlation: close to 1.0 across all evaluation dimensions | Improved |
| compare kinetic metrics against appearance-based VBench axes | HumanScore Kinetic vs Kinematic · VBench correlation | Spearman correlation: Kinematic 0.927 vs Kinetic 0.349 (imaging quality) | Mixed |
| increase motion difficulty from easy to hard | all video generators · HumanScore difficulty breakdown | HumanScore | Hurt |
| generated videos versus real videos | best video generators · HumanScore leaderboard | Overall HumanScore: 94.3 (real) vs 91.1 (best generated) | Hurt |
Surprising findings
- Real videos do not reach perfect scores, with kinematic average 89.4, because monocular 3D reconstruction is itself noisy.
- Kinematic correctness correlates strongly with VBench imaging quality (0.927) while kinetic correctness correlates weakly (0.349), suggesting appearance metrics miss kinetic realism.
Limitations
- Relies on external pose/mesh estimators (HADM, PromptHMR, MeTRAbs) whose errors propagate into scores, especially kinematics.
- Single-person videos only; multi-person extension is untested.
- Metric weights (alpha, beta, gamma) and tolerances are user-defined, though robustness analysis is provided.
Experimental setup
Datasets: Kinetics-700 real videos collected from YouTube, Bilibili, YouTube-8M
Benchmarks: HumanScore
Metrics: HumanScore (0-100 per dimension and overall) Spearman rank correlation with human preference Spearman correlation with VBench axes
Baselines: VBench Wan 2.2 CogVideoX-5B HunyuanVideo 1.5 Kandinsky 5.0 pro Sora-2 Veo 3.1 fast KlingAI 2.5 Turbo Pro Seedance 1.0 Pro fast Hailuo 02
Builds on: VBench HADM PromptHMR MeTRAbs Kinetics-700
Closest papers by meaning
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.91)
- Aligning Human Motion Generation with Human Perceptions (ICLR 2025 · similarity 0.90)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.89)
- OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation (CVPR 2025 · similarity 0.89)
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.88)
- MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
