Research map / Video / Video generation
On the Content Bias in Frechet Video Distance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why Fréchet Video Distance (FVD) favors per-frame quality over temporal realism, showing its I3D features are content-biased and that self-supervised VideoMAE-v2 features reduce this bias.
Problem
FVD, the standard metric for video generation, sometimes contradicts human judgment by overweighting per-frame image quality relative to motion/temporal realism.
Approach
Decouples spatial and spatiotemporal distortions to measure FVD's temporal sensitivity, probes a "temporal perceptual null space" by resampling frozen videos, and traces the bias to the supervised I3D feature extractor; replacing I3D with self-supervised VideoMAE-v2 features markedly increases temporal sensitivity.
Key results
- Under spatiotemporal vs spatial distortion with matched FID, I3D-based FVD rises only about 0-36% (e.g., 3% on FaceForensics), while VideoMAE-v2 features show roughly 5x larger FVD increase under elastic transformation.
- Resampling frozen (motion-free) generated videos reduces I3D-based FVD by 40%-70%, whereas VideoMAE-v2-SSv2 FVD decreases by a more moderate 25%-60% in the reported range (with some models around 5-13%).
- In long-video case studies, I3D FVD favors artifact-laden StyleGAN-V-LSTM and periodic DIGAN outputs, while VideoMAE-v2 features agree with human judgment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace I3D features with VideoMAE-v2 features | FVD · Video distortions across multiple datasets | temporal sensitivity of FVD: increases five times more | Helped |
| Use VideoMAE-v2-SSv2 features instead of I3D features | FVD · Video distortions across multiple datasets | temporal sensitivity of FVD: at least threefold | Helped |
| Select generated videos by minimizing weighted FVD, without adding motion | FVD with I3D features · UCF-101 | FVD: 1303.13 -> 715.96 (−45.1%) | Helped |
| Select generated videos by minimizing weighted FVD, without adding motion | FVD with I3D features · Sky Time-lapse | FVD: 230.64 -> 115.55 (−49.9%) | Helped |
| Replace I3D features with VideoMAE-v2-SSv2 features during resampling | FVD · UCF-101 | FVD decrease after resampling frozen videos: −45.1% vs. −9.7% | Hurt |
| Replace default StyleGAN-v motion codes with LSTM codes (ablation) | StyleGAN-v · Sky Time-lapse, 128 frames | FVD128: 190.82 -> 172.71(−18.11) | Helped |
| Replace default StyleGAN-v motion codes with LSTM codes (ablation) | StyleGAN-v · Sky Time-lapse, 128 frames | FVD128: 332.80 -> 616.74(+283.94) | Hurt |
| Evaluate DIGAN's extrapolated frames 128–144 instead of frames 0–16 | DIGAN · Sky Time-lapse | FVD: 155.58 -> 141.82(−8.84%) | Helped |
| Evaluate DIGAN's extrapolated frames 128–144 instead of frames 0–16 | DIGAN · Sky Time-lapse | FVD: 133.37 -> 150.61(+12.9%) | Hurt |
| Apply different spatial distortions to each frame, introducing temporal inconsistency | I3D-based FVD · FaceForensics | FVD: 354.49 -> 367.35(+3.6%) | Hurt |
Surprising findings
- FVD with I3D features sometimes favors the video set with more severe temporal corruption (lower FVD for the spatiotemporal distortion than the spatial one).
- Reducing FVD by up to half is possible by resampling videos with no motion at all on Sky Time-lapse.
Limitations
- Analysis limited to 224x224 resized, square videos; megapixel and non-square generation not covered.
- VideoMAE-v2 features are costly due to quadratic attention cost, limiting longer video evaluation.
- Could not train VideoMAE-v2 ViT from scratch with supervised objective due to compute, so the origin of the bias is partly inferred from smaller comparisons.
Experimental setup
Datasets: Kinetics-400 Something-Something-v2 UCF-101 Sky Time-lapse FaceForensics Taichi-HD Kinetics-710
Benchmarks: UCF-101 Sky Time-lapse FaceForensics Taichi-HD SSv2 Kinetics-400
Metrics: FVD FID FVD change ratio FVD decrease by resampling
Baselines: I3D-based FVD DIGAN TATS StyleGAN-V PVDM
Builds on: FVD FID Kynkäänniemi resampling analysis
Closest papers by meaning
- Direct Motion Models for Assessing Generated Videos (ICML 2025 · similarity 0.87)
- STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models (ICLR 2024 · similarity 0.87)
- Representation Fréchet Loss for Visual Generation (NeurIPS 2026 · similarity 0.86)
- Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs (CVPR 2025 · similarity 0.86)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.85)
- Rethinking FID: Towards a Better Evaluation Metric for Image Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.