Research map / Video / Video segmentation and tracking
ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExtDM, a diffusion video prediction method that extrapolates motion cues (optical flow and occlusion maps) from present to future using a layered distribution adaptor and a spatiotemporal-window 3D U-Net, achieving faster and more accurate short- and long-term prediction.
Problem
Video prediction over long horizons suffers from uncertainty and a gap between present and future; diffusion-based methods repeatedly run heavy 3D spatiotemporal U-Nets at full resolution, making them slow.
Approach
Casts video prediction as extrapolating compact motion cues (flow and occlusion maps) from present to future via a Gaussian-distribution-based layered adaptor, then uses a sparse spatiotemporal-window U-Net in a diffusion model to refine them and a motion autoencoder to reconstruct frames.
Key results
- On KTH (u=10, v=40), ExtDM-K4 reaches FVD 221.4 and PSNR 27.91 versus 276.6 FVD for MCVD-c, at 45.28 FPS versus 6.35.
- On BAIR (u=2, v=28), ExtDM-K3 achieves PSNR 18.83 and FVD 114.7, improving over the best MCVD variant (FVD 118.4).
- The paper reports ExtDM runs 7.51 times faster than prior diffusion-based video prediction methods on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ExtDM-K4 instead of LFDM | KTH video prediction · KTH (u=10, v=40) | PSNR: 26.41 -> 27.91 | Helped |
| use ExtDM-K4 instead of MCVD-c | KTH video prediction · KTH (u=10, v=40) | FVD: 276.6 -> 221.4 | Helped |
| use ExtDM-K4 instead of MCVD-sp | BAIR video prediction · BAIR (u=2, v=28) | PSNR: 17.70 -> 18.74 | Helped |
| use ExtDM-K4 instead of MCVD-cpf | BAIR video prediction · BAIR (u=2, v=28) | FVD: 118.4 -> 102.8 | Helped |
| use ExtDM-K4 instead of MCVD-s | Cityscapes video prediction · Cityscapes (u=2, v=28) | PSNR: 22.50 -> 22.84 | Helped |
| use ExtDM-K4 instead of MCVD-c | Cityscapes video prediction · Cityscapes (u=2, v=28) | FVD: 141.3 -> 121.3 | Helped |
| use ExtDM-K4 instead of RVD | SMMNIST video prediction · SMMNIST (u=10, v=10) | PSNR: 18.56 -> 19.59 | Helped |
| use ExtDM-K4 instead of RVD | SMMNIST video prediction · SMMNIST (u=10, v=10) | FVD: 17.84 -> 11.11 | Helped |
| use ExtDM-K2 instead of MCVD-cp | UCF-101 video prediction · UCF-101 (u=4, v=12) | PSNR: 21.82 -> 23.89 | Helped |
| use ExtDM-K2 instead of RaMViD | UCF-101 video prediction · UCF-101 (u=4, v=12) | FVD: 396.7 -> 394.1 | Helped |
| add compressed-space motion representation (ablation) | ExtDM · KTH (u=10, v=40) | PSNR: 23.43 -> 25.39 | Helped |
| add spatiotemporal-window attention (ablation) | ExtDM · KTH (u=10, v=40) | PSNR: 27.12 -> 27.91 | Helped |
Limitations
- Multi-layer LDA variants increase computation and reduce speed relative to single-layer variant.
- Performance depends on quality of estimated optical flow and occlusion maps; ground-truth motion gives higher reconstruction quality, indicating estimation error.
Experimental setup
Datasets: KTH BAIR Cityscapes SMMNIST UCF-101
Benchmarks: KTH BAIR Cityscapes SMMNIST UCF-101
Metrics: SSIM PSNR LPIPS FVD FPS
Baselines: U-ViT DiT RaMViD LVDM RVD VIDM LFDM MCVD
Builds on: Video Diffusion Models (VDM) MCVD LFDM LVDM Swin-style shifted window attention
Closest papers by meaning
- STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · similarity 0.87)
- RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers (ICML 2025 · similarity 0.85)
- Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data (ICLR 2025 · similarity 0.85)
- STDD: Spatio-Temporal Dual Diffusion for Video Generation (CVPR 2025 · similarity 0.85)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.85)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.