Research map / Video / Video generation
Parameterized Stripe Attention for Efficient Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSA, a parameterized periodic-stripe sparse attention with a single FA3-style Hopper kernel and an offline per-head search, speeding up HunyuanVideo and Wan 2.1 video DiTs over FlashAttention-3.
Problem
Full spatio-temporal attention in video DiTs dominates inference latency; existing sparse attention either uses predefined masks with limited pattern coverage or runtime-determined masks with overheads that hurt hardware efficiency.
Approach
Video DiT attention shows periodic diagonal stripes (from spatio-temporal locality plus RoPE-3D cosine periodicity); a six-parameter mask M(Tw,To,Ts,Sw,So,Ss) covers these patterns, is implemented in one block-sparse Hopper kernel, and per-head masks are chosen offline by maximizing sparsity under an L2 error threshold.
Key results
- On HunyuanVideo 768x1280 at 60.33% sparsity, PSA reaches 1.57x E2E speedup over FA3 with PSNR 27.20 vs SVG2 25.80 at 1.43x.
- Kernel MFU reaches 61.21% on HunyuanVideo 768x1280 and 53.68% on Wan 2.1 14B, versus 38.18% for SVG2 kernels.
- On Wan 2.1 14B at 57.57% sparsity, PSA achieves 1.37x E2E speedup over FA3, the lowest E2E gain among the reported setups.
Surprising findings
- The paper reports that SVG2 and SVG are not clearly better on PSNR for E2E at some sparsity levels; the E2E speedup gap over SVG2 on Wan 2.1 is small (1.37x vs 1.31x).
Limitations
- Only evaluated on video generation models; transfer to language or other domains is uncertain.
- Offline search takes 2.5 hours on 8 H100 GPUs.
- Quality degrades with sparsity (PSNR falls to about 30 at 60% sparsity on HunyuanVideo 720p).
Experimental setup
Datasets: VBench prompts
Benchmarks: VBench
Metrics: E2E latency OC TS MSE PSNR SSIM LPIPS TFLOPs MFU
Baselines: FlashAttention-2 FlashAttention-3 SVG SVG2 STA
Builds on: FlashAttention-3 ThunderKittens RoPE SVG STA
Closest papers by meaning
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.91)
- Fast Video Generation with Sliding Tile Attention (ICML 2025 · similarity 0.90)
- PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers (NeurIPS 2026 · similarity 0.90)
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.90)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.90)
- MonarchRT: Efficient Attention for Real-Time Video Generation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.