Research map / Video / Video generation
VDT: General-purpose Video Diffusion Transformers via Mask Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VDT, a pure-transformer video diffusion model with factorized temporal/spatial attention and a unified spatial-temporal mask modeling scheme that handles unconditional generation, prediction, interpolation, and completion.
Problem
Diffusion-based video generation has mostly relied on U-Nets and specialized modules per task; it is unclear whether transformers can serve as the backbone and unify diverse video generation tasks (unconditional generation, prediction, interpolation, animation, completion) in one model.
Approach
Use a pure transformer denoiser with separate temporal and spatial attention in each block, condition by simple token concatenation of observed latent frames, and unify tasks by a binary spatial-temporal mask that selects real conditional tokens versus noise during training.
Key results
- VDT-L reaches FVD 225.7 on UCF101 unconditional generation, and VDT-S reaches 425.6.
- On Cityscapes prediction (2 to 28 frames), VDT achieves FVD 142.3 and SSIM 0.880, comparable to MCVD-concat in FVD (141.4) and better in SSIM.
- On Physion-Collide VQA, VDT reaches 65.3% accuracy, the best among scene-centric methods (best baseline 63.1%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use token concatenation instead of adaptive layer normalization or cross-attention for conditioning (ablation) | VDT · Physion prediction | FVD: 270.8 -> 129.1; 134.9 -> 129.1 | Helped |
| Use token concatenation instead of adaptive layer normalization or cross-attention for conditioning (ablation) | VDT · Physion prediction | SSIM: 0.6247 -> 0.8718; 0.8523 -> 0.8718 | Helped |
| Use token concatenation instead of adaptive layer normalization (ablation) | VDT · Physion prediction | PSNR: 16.8 -> 30.2 | Helped |
| Use token concatenation instead of cross-attention (ablation) | VDT · Physion prediction | PSNR: 28.6 -> 30.2 | Helped |
| Spatial pretrain before joint training (ablation) | VDT-S · UCF101 unconditional generation | FVD: 451.9 vs. 431.7 | Helped |
| Spatial pretrain before joint training (ablation) | VDT-S · UCF101 unconditional generation | Time: 11.2 vs. 14.4 | Helped |
| Increase joint training from 40k to 80k steps (ablation) | VDT-S · UCF101 unconditional generation | FVD: 554.8 -> 451.9 | Helped |
| Increase joint training from 80k to 120k steps (ablation) | VDT-S · UCF101 unconditional generation | FVD: 451.9 -> 425.6 | Helped |
| Scale from VDT-S to VDT-L (ablation) | VDT · UCF101 unconditional generation | FVD: 425.6 -> 225.7 | Helped |
Surprising findings
- Token concatenation converged fastest and outperformed adaptive layer norm and cross-attention for conditioning, despite its simplicity.
Limitations
- Could not pretrain on large-scale image or video datasets due to GPU limits.
- Architecture is a standard transformer with no significant modifications; architectural influence left to future work.
- Text conditioning not explored.
Experimental setup
Datasets: UCF101 TaiChi Sky Time-Lapse Cityscapes Physion
Benchmarks: UCF101 TaiChi Sky Time-Lapse Cityscapes Physion
Metrics: FVD SSIM PSNR VQA accuracy
Baselines: MCVD VDM DIGAN VideoFusion SVG-LP vRNN 1L Hier-vRNN GHVAE SlotFormer PRIN pVGG-lstm pDEIT-lstm
Builds on: Latent Diffusion Models DiT ViT BEiT MCVD
Closest papers by meaning
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.89)
- Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation (CVPR 2025 · similarity 0.88)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.87)
- Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers (ICML 2025 · similarity 0.87)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.