Research map / Video / Video segmentation and tracking
STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STDiff, a stochastic video prediction model that uses a neural SDE over motion features (from frame differences via ConvGRU) plus an autoregressive image diffusion model, enabling continuous-time frame prediction with high diversity.
Problem
Stochastic video frame prediction is hard because future outcomes are uncertain; existing models have limited stochastic expressiveness along time and cannot predict at arbitrary (continuous) frame rates with good diversity.
Approach
Decompose video into motion and content; model temporal motion stochasticity with a neural SDE that can be integrated at arbitrary time points, and generate each frame with a conditional image diffusion model conditioned on the predicted motion feature and previous frame, so both spatial and temporal latents are SDE-described.
Key results
- On SMMNIST (5→10), STDiff achieves FVD 14.93, beating the next best MCVD-spatin at 23.86.
- On KITTI (4→5), STDiff reports FVD 51.39 and LPIPS 114.6, versus NPVP at 134.69 FVD and 279.0 LPIPS.
- On KTH (10→20), STDiff reports iLPIPS 25.08 and FVD 89.67, with iLPIPS about 50 times NPVP's on KTH per the paper's claim.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STDiff instead of prior methods | SMMNIST, 5 → 10 · SMMNIST | FVD: 23.86 -> 14.93 | Helped |
| use STDiff instead of prior methods | BAIR, 2 → 28 · BAIR | FVD: 93.6 -> 88.1 | Helped |
| use STDiff instead of prior methods | KITTI, 4 → 5 · KITTI | LPIPS: 260.5 -> 114.6 | Helped |
| use STDiff instead of prior methods | Cityscapes, 2 → 28 · Cityscapes | FVD: 141.31 -> 107.31 | Helped |
| replace the SDE-based motion predictor with an ODE-based motion predictor (ablation) | STDiff · KTH, 10 → 20 | iLPIPS: 25.08 -> 15.60 | Hurt |
| replace the SDE-based motion predictor with an ODE-based motion predictor (ablation) | STDiff · KTH, 10 → 20 | FVD: 89.67 -> 90.68 | Hurt |
| replace the SDE-based motion predictor with an ODE-based motion predictor (ablation) | STDiff · SMMNIST, 5 → 10 | iLPIPS: 136.27 -> 91.07 | Hurt |
| replace the SDE-based motion predictor with an ODE-based motion predictor (ablation) | STDiff · SMMNIST, 5 → 10 | FVD: 14.93 -> 46.62 | Hurt |
| use STDiff instead of NPVP for continuous prediction | BAIR, 2 → 28, 2× fps · BAIR | FVD: 1159.14 -> 122.85 | Helped |
| use STDiff instead of NPVP for continuous prediction | KITTI, 4 → 5, 2× fps · KITTI | LPIPS: 313.94 -> 160.92 | Helped |
Surprising findings
- On SMMNIST, STDiff's SSIM (0.748) is below several baselines, and on KITTI its SSIM (0.54) trails DMVFN (0.71).
Limitations
- Lower SSIM than several baselines on some datasets (e.g., SMMNIST, KITTI).
- Diversity and FVD results depend on protocol choices (e.g., 10 vs 100 samples, reverse diffusion steps of 100).
Experimental setup
Datasets: KITTI Cityscapes KTH BAIR SMMNIST
Benchmarks: KITTI Cityscapes KTH BAIR SMMNIST
Metrics: FVD SSIM LPIPS iLPIPS
Baselines: SVG-LP Hier-VRNN MCVD NPVP SAVP STMFANet VPTR-NAR FitVid PredNet Voxel Flow SADM DMVFN
Builds on: denoising diffusion probabilistic models neural SDE MCVD SPADE Vid-ODE
Closest papers by meaning
- STDD: Spatio-Temporal Dual Diffusion for Video Generation (CVPR 2025 · similarity 0.91)
- Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes (CVPR 2024 · similarity 0.88)
- ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction (CVPR 2024 · similarity 0.87)
- Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · similarity 0.87)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.86)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.