Research map / Video / Video generation
STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STREAM, a video generative model evaluation metric that uses per-frame image embeddings with FFT along time to separately score temporal naturalness (STREAM-T) and spatial fidelity/diversity (STREAM-F/STREAM-D), unconstrained by video length.
Problem
Existing video metrics (FVD, VIS) are adaptations of image metrics that emphasize spatial quality over temporal naturalness, are limited to 16-frame clips by their embedding network, are unstable, and give only single composite scores, so they cannot diagnose video generator weaknesses.
Approach
Use an image embedding network on each frame, then apply FFT along the time axis: STREAM-T compares skewness of power-law-fitted amplitude distributions across feature dimensions between real and fake videos (correlation of histograms), while STREAM-S applies improved precision/recall to the mean (zero-frequency) amplitude to get bounded fidelity and diversity scores.
Key results
- On UCF-101 16-frame generation, STREAM-T reaches 0.9843 for PVDM and 0.9832 for TATS, while STREAM-F is highest for TATS at 0.9120.
- Human study with 81 raters gives Spearman correlations of 0.9 (realism) and 0.6 (temporal naturalness) for STREAM versus 0.7 and 0.5 for FVD.
- Under random translation on toy data, FVD rises to 822, over three times its response to fully static stop scenes, indicating FVD over-weights spatial quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STREAM-T and STREAM-S components to evaluate realism and temporal naturalness separately | Video generative model evaluation · UCF-101, 16-frame generations | Spearman correlation with human judgments: STREAM: 0.9 for realism and 0.6 for temporal naturalness; FVD: 0.7 and 0.5 | Helped |
| Generate 128 frames instead of 16 frames | TATS-base, MoCoGAN, DIGAN, and MeBT · UCF-101 video generation | STREAM-T | Hurt |
| Use a video embedding network instead of an image embedding network (ablation) | STREAM-T · UCF-101 visual and temporal distortion tests | Temporal-flow sensitivity | Hurt |
| Include the mean-amplitude signal in STREAM-T (ablation) | STREAM-T · Visual distortion and global-swap tests | Temporal naturalness evaluation | Mixed |
| Vary STREAM-T histogram bin size across tested settings (ablation) | STREAM-T · Visual and temporal distortion tests | Evaluation consistency | No clear effect |
| Use an extremely small histogram bin size (ablation) | STREAM-T · Temporal distortion tests | Evaluation consistency | Hurt |
| Set histogram bin size to 50 (ablation) | STREAM-T · Visual and temporal distortion tests | Evaluation consistency | Helped |
| Increase evaluation data size to approximately 2,000 (ablation) | STREAM-T · Local swap, global swap, and random translation | Score consistency | Helped |
| Reduce fake-video FPS relative to real-video FPS | STREAM-T · FPS variation experiment | STREAM-T: 0.996 ($\pm$0.000) -> 0.352 ($\pm$0.004) | Hurt |
| Reverse only partial video sequences with increasing intensity | STREAM-T · UCF-101, partial reverse | STREAM-T: 0.9948 -> 0.7916 | Hurt |
Surprising findings
- Video embedding network MVD2 mixes visual and temporal information, so STREAM-T did not behave as intended with it, motivating the use of an image embedding network.
- FVD increased from 14.38 to 36.07 when videos were only horizontally flipped, even though flipping preserves realism.
Limitations
- STREAM-T is agnostic to time direction, so fully reversed videos may not be detected by any metric.
- Requires around 2,000 samples for consistent STREAM-T scores.
- Diversity is hard to assess with human evaluation, so diversity was not validated against humans.
Experimental setup
Datasets: CATER UCF-101 Kinetics-600 Taichi
Benchmarks: CATER UCF-101 Kinetics-600 Taichi
Metrics: STREAM-T STREAM-F STREAM-D FVD VIS sFVD sVIS Spearman's rank correlation
Baselines: FVD VIS sFVD sVIS
Builds on: FVD Improved precision and recall Inception Score Fréchet Inception Distance
Closest papers by meaning
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.88)
- Direct Motion Models for Assessing Generated Videos (ICML 2025 · similarity 0.88)
- On the Content Bias in Frechet Video Distance (CVPR 2024 · similarity 0.87)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.86)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.86)
- Rethinking FID: Towards a Better Evaluation Metric for Image Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.