Research map / Video / Video generation
STDD: Spatio-Temporal Dual Diffusion for Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spatio-Temporal Dual Diffusion (STDD), which extends DDPM with a recursive temporal term so each frame's diffusion state depends on the previous frame, with a DDIM-style accelerated sampler, evaluated on video prediction and generation.
Problem
Existing video diffusion models treat frames as independent diffusion processes with independent priors, breaking temporal correlation and hurting frame-to-frame continuity.
Approach
Extends the standard diffusion chain with a temporal Markov term that mixes the current frame's latent with the previous frame's latent (weight rho), yielding a closed-form dual diffusion process g(t,i) that accumulates history frames; derives a reverse process and a DDIM-style accelerated sampler, with the video noise prior tied across frames.
Key results
- On MHAD at 128x128, STDD reaches FVD 195 and cFVD 411, versus 214 FVD for LFDM (second best).
- On UCF101, STDD reports FVD 258 versus 333 for Latte+IMG and 372 for LVDM; on Sky Time-Lapse it reports 42.1 versus 42.7 for Latte+IMG.
- Accelerated sampling uses 250 steps; inference takes 23s versus 20s for Latte on an RTX 3090 with 250 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STDD instead of LFDM | Video prediction · MHAD, 128 × 128 | FVD: 214 -> 195 | Helped |
| Use STDD instead of LFDM | Video prediction · MHAD, 128 × 128 | cFVD: 426 -> 411 | Helped |
| Use STDD instead of Latte+IMG | Video generation · UCF101 | FVD: 333 -> 258 | Helped |
| Use STDD instead of Latte+IMG | Video generation · Sky Time-Lapse | FVD: 42.7 -> 42.1 | Helped |
| Set ρ to 1 instead of 0.999 (ablation) | STDD · MHAD, 64 × 64 | FVD: 133 -> 157 | Hurt |
| Set ρ to 0.9999 instead of 0.999 (ablation) | STDD · MHAD, 64 × 64 | FVD: 133 -> 149 | Hurt |
| Set ρ to 0.995 instead of 0.999 (ablation) | STDD · MHAD, 64 × 64 | FVD: 133 -> 243 | Hurt |
| Set ρ to 0.99 instead of 0.999 (ablation) | STDD · MHAD, 64 × 64 | FVD: 133 -> 388 | Hurt |
Surprising findings
- Its text-to-video results are shown only qualitatively, with no quantitative FVD reported on WebVid-10M.
Limitations
- Text-to-video evaluation is qualitative only, without quantitative metrics.
- Performance depends on the choice of rho; too small a rho loses current-frame information, and the approximation requires 1-rho < 0.01.
- Model size (868M) is larger than LVDM (437M) and Latte (673M).
Experimental setup
Datasets: UCF101 Sky Time-Lapse MHAD WebVid-10M
Benchmarks: MHAD UCF101 Sky Time-Lapse
Metrics: FVD cFVD
Baselines: ImaGINator VDM LDM LFDM VideoGPT MoCoGAN-HD DIGAN StyleGAN-V PVDM LVDM Latte+IMG TATS
Builds on: DDPM DDIM Latte LDM
Closest papers by meaning
- STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · similarity 0.91)
- LongDiff: Training-Free Long Video Generation in One Go (CVPR 2025 · similarity 0.88)
- Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data (ICLR 2025 · similarity 0.88)
- Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling (CVPR 2025 · similarity 0.88)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.88)
- Solving Video Inverse Problems Using Image Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.