Research map / Video / Video generation
Direct Motion Models for Assessing Generated Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRAJAN, a transformer autoencoder over BootsTAPIR point tracks whose latent space and reconstruction error serve as motion-focused metrics for generated videos at distribution, pair, and single-video levels.
Problem
Existing video metrics such as FVD are biased toward frame-level appearance and fail to capture motion plausibility; they also require reference distributions and do not easily handle video pairs or single videos.
Approach
A Perceiver-style trajectory autoencoder trained to reconstruct held-out point tracks from random query points over a set of input tracks, producing an order-invariant, occlusion-aware motion latent; its latent distances and per-video reconstruction error (Average Jaccard) act as motion-consistency metrics independent of appearance.
Key results
- TRAJAN is more than three times more sensitive to synthetic temporal distortions than the best alternative under Fréchet distance on UCF101.
- On the human-rated EvalCrafter subset, TRAJAN achieves Spearman 0.29 for motion consistency and 0.27 for realism, outperforming RAFT warp (0.28 / 0.25) and VideoMAE (-0.00 / 0.04).
- On the original VideoPhy physical-consistency labels, TRAJAN reaches competitive ROC-AUC without any fine-tuning on human labels, performing strongly relative to VideoCon which is fine-tuned on those labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TRAJAN instead of RAFT - Warp | Automated motion-consistency scoring · EvalCrafter | Spearman correlation with human motion-consistency ratings: 0.28 -> 0.29 | Helped |
| Use TRAJAN instead of RAFT - Warp | Automated appearance-consistency scoring · EvalCrafter | Spearman correlation with human appearance-consistency ratings: 0.27 -> 0.29 | Helped |
| Use TRAJAN instead of RAFT - Warp | Automated realism scoring · EvalCrafter | Spearman correlation with human realism ratings: 0.25 -> 0.27 | Helped |
| Use TRAJAN instead of I3D - Entropy | Automated interactions scoring · EvalCrafter | Spearman correlation with human interactions ratings: 0.03 -> 0.19 | Helped |
| Use TRAJAN instead of RAFT - Warp | Automated motion-consistency scoring · VideoPhy | Spearman correlation with human motion-consistency ratings: 0.20 -> 0.25 | Helped |
| Use TRAJAN instead of RAFT - Warp | Automated appearance-consistency scoring · VideoPhy | Spearman correlation with human appearance-consistency ratings: 0.26 -> 0.32 | Helped |
| Use TRAJAN instead of RAFT - Warp | Automated realism scoring · VideoPhy | Spearman correlation with human realism ratings: 0.18 -> 0.29 | Helped |
| Use TRAJAN instead of I3D - Entropy | Automated interactions scoring · VideoPhy | Spearman correlation with human interactions ratings: 0.09 -> 0.09 | No clear effect |
Surprising findings
- Human judgements on the original EvalCrafter dataset were largely driven by overall motion magnitude, with faster videos rated worse regardless of realism.
- Human raters were inconsistent with one another (inter-rater sigma around 0.5), and raters seemed to conflate semantic realism with motion consistency.
Limitations
- No automated metric captures human ratings for physically implausible but smooth motions (e.g., glass collapsing into a table).
- Human raters show low consistency with each other, limiting the reliability of the human gold standard.
- Correlations with human judgements are modest (Spearman roughly 0.2-0.3) even for the best metric.
Experimental setup
Datasets: UCF101 VideoPhy EvalCrafter Kinetics-600 internal 15M-video corpus
Benchmarks: UCF101 synthetic corruptions VideoPhy EvalCrafter WALT generated vs real pairs
Metrics: Fréchet distance sensitivity Spearman rank correlation ROC-AUC PSNR/SSIM correlation
Baselines: VideoMAE v2 I3D motion histograms RAFT warp error MooG VideoCon
Builds on: BootsTAPIR Perceiver FVD TAP-Vid Average Jaccard FVMD motion histograms
Closest papers by meaning
- 3DSPA: A 3D Semantic Point Autoencoder for Evaluating Video Realism (NeurIPS 2026 · similarity 0.88)
- STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models (ICLR 2024 · similarity 0.88)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.88)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.87)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- On the Content Bias in Frechet Video Distance (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.