Research map / Video / Video generation
Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparse VideoGen (SVG), a training-free sparse attention framework for video diffusion transformers that classifies heads as spatial or temporal via online profiling and uses a layout transformation for hardware-efficient sparse attention.
Problem
Video diffusion transformers use 3D full attention whose quadratic cost makes generation extremely slow (e.g., HunyuanVideo takes ~40 minutes for a 5-second 720p video on an A100), and existing LLM-style sparse attention does not match video's spatial/temporal sparsity patterns.
Approach
Attention heads in video DiTs fall into spatial heads (block-wise intra-frame attention) and temporal heads (slash-wise same-position attention across frames); an online profiling step on ~1% of query rows picks the lower-error pattern per head and step, and a token-major to frame-major layout transformation makes temporal sparsity contiguous for tensor cores.
Key results
- On HunyuanVideo, SVG with FP8 reaches 2.33x end-to-end speedup (2253s to 968s) with PSNR 29.45.
- On CogVideoX-v1.5-T2V, SVG achieves 2.28x speedup with PSNR 29.989, versus 22.45-23.80 for baselines.
- Profiling only 1% of query rows reaches PSNR 31.118 on CogVideoX-v1.5-I2V, close to the 100% oracle at 31.324, with about 3% runtime overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive spatial/temporal sparse attention instead of spatial-only attention | CogVideoX-v1.5-I2V · CogVideoX-v1.5-I2V (720p, 10s, 80 frames) | PSNR: 24.591 -> 28.165 | Helped |
| Use adaptive spatial/temporal sparse attention instead of spatial-only attention | CogVideoX-v1.5-I2V · CogVideoX-v1.5-I2V (720p, 10s, 80 frames) | speedup: 1.56x -> 2.23x | Helped |
| Use adaptive spatial/temporal sparse attention instead of temporal-only attention | CogVideoX-v1.5-T2V · CogVideoX-v1.5-T2V (720p, 10s, 80 frames) | PSNR: 23.804 -> 29.989 | Helped |
| Use adaptive spatial/temporal sparse attention instead of temporal-only attention | HunyuanVideo · HunyuanVideo-T2V (720p, 5.33s, 128 frames) | PSNR: 25.851 -> 29.546 | Helped |
| Add FP8 quantization to Sparse VideoGen | Sparse VideoGen on HunyuanVideo · HunyuanVideo-T2V | speedup: 1.92x -> 2.33x | Helped |
| Add FP8 quantization to Sparse VideoGen | Sparse VideoGen on HunyuanVideo · HunyuanVideo-T2V | PSNR: 29.546 -> 29.452 | Hurt |
| Use 1% profiling rather than 0.1% profiling (ablation) | Sparse VideoGen · CogVideoX-v1.5-I2V | PSNR: 30.791 -> 31.118 | Helped |
| Increase profiling from 1% to 5% (ablation) | Sparse VideoGen · CogVideoX-v1.5-I2V | PSNR: 31.118 -> 31.008 | Hurt |
| Use full profiling (100%) instead of 1% profiling (ablation) | Sparse VideoGen · CogVideoX-v1.5-I2V | PSNR: 31.118 -> 31.324 | Helped |
| Increase sparsity from 0.13 to 0.18 (ablation) | Sparse VideoGen · HunyuanVideo VBench subset | LPIPS: 0.154 -> 0.135 | Helped |
| Increase sparsity from 0.18 to 0.35 (ablation) | Sparse VideoGen · HunyuanVideo VBench subset | LPIPS: 0.135 -> 0.141 | Hurt |
| Add hardware-efficient layout transformation for temporal attention (ablation) | Sparse VideoGen sparse-attention kernel · HunyuanVideo configuration, sparsity 10% | speedup: 1.7× faster; 3.63× improvement | Helped |
Surprising findings
- Temporal-only attention alone achieves higher PSNR than spatial-only DiTFastAttn on HunyuanVideo (25.85 vs 21.42), suggesting temporal heads matter more for quality there.
Limitations
- Sparsity ratios (c_s, c_t) are fixed by hand; adaptive sparsity control is left for future work.
- FP8 attention not applied to CogVideoX-v1.5 because head dimension 64 yields no on-GPU speedup.
- Quality evaluated mainly with PSNR/SSIM/LPIPS against dense outputs and VBench subsets; the paper attributes spatial/temporal heads partly to the slow-motion nature of training data (hypothesis).
Experimental setup
Datasets: VBench Penguin Video Benchmark
Benchmarks: VBench Penguin Video Benchmark
Metrics: PSNR SSIM LPIPS ImageQuality SubjectConsistency FLOPS latency speedup
Baselines: DiTFastAttn MInference PAB Temporal-only attention
Builds on: DiTFastAttn MInference PAB FlashAttention FlashInfer
Closest papers by meaning
- Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation (NeurIPS 2025 · similarity 0.93)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.92)
- MonarchRT: Efficient Attention for Real-Time Video Generation (NeurIPS 2026 · similarity 0.91)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.90)
- HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion (NeurIPS 2026 · similarity 0.90)
- iCATS: Fast Video Generation via Interaction-Aware Sparse Attention and Timestep-Adaptive Sparsity (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.