Research map / Architectures and efficiency / Attention and transformers

Faster Video Diffusion with Trainable Sparse Attention

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VSA, a trainable hardware-aligned hierarchical sparse attention for video diffusion transformers that uses a coarse cube-level stage to select critical tiles for a block-sparse fine stage, and validates it via pretraining ablations/scaling, Wan2.1 retrofitting, and sparse distillation.

Problem

Full 3D attention in video diffusion transformers is quadratic in sequence length, making both training and inference expensive; post-hoc sparse methods create train-test mismatch and identifying critical tokens cheaply while respecting block-sparse GPU layouts is hard.

Approach

Mean-pool tokens into 4x4x4 cubes, run a coarse cube-to-cube dense attention to select top-K critical cubes and output a coarse context, run block-sparse token-level fine attention only over selected cubes, and combine both via learned gates; trained end-to-end with a single kernel co-designed so each cube maps to one GPU tile.

Key results

• VSA matches full-attention diffusion loss at 87.5% sparsity while cutting training FLOPS by 2.53x and reaching 85% of FlashAttention-3 MFU.
• Retrofitting Wan2.1-1.3B with VSA speeds attention 6x and reduces end-to-end generation from 31s to 18s with comparable VBench quality; for Wan2.1-14B, generation time falls from 1274s to 576s.
• Sparse distillation (VSA in DMD2 student) on Wan-1.3B achieves 50.9x speedup with no quality drop in the reported study.

Measured effects reported in the paper

Surprising findings

• Optimal top-K did not monotonically increase with sequence length: K=32 underperformed K=16 at 61440 tokens under a fixed small compute budget but outperformed it at higher compute.
• Purposely injected local attention priors gave negligible benefit, and mean pooling outperformed max and convolutional pooling, with conv pooling causing training instability.

Limitations

• Fixed cube size (4,4,4) requires latent dimensions divisible by 4, constraining resolutions.
• Coarse-stage top-K selection is not fused into FlashAttention and adds runtime overhead at shorter sequence lengths (about 14% of runtime at high sparsity).
• Optimal sparsity (top-K) depends on sequence length and compute budget and lacks a predictive rule; experiments mainly at 16K sequence length for pretraining.

Experimental setup

Datasets: Vchitect-T2V-Dataverse Synthetic videos generated by Wan-14B MovieGen prompts

Benchmarks: VBench Diffusion loss on pretraining ablations

Metrics: diffusion loss VBench score attention FLOPS training FLOPS MFU end-to-end generation latency critical token prediction accuracy human preference

Baselines: Full attention Sliding Tile Attention Sparge Attention SVG FlexAttention DSV MoBA NSA BiFormer

Builds on: Flash Attention Native Sparse Attention (NSA) MoBA Wan2.1 DMD2

Closest papers by meaning

• Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.94)
• Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.92)
• PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers (NeurIPS 2026 · similarity 0.91)
• VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.91)
• VORTA: Efficient Video Diffusion via Routing Sparse Attention (NeurIPS 2025 · similarity 0.91)
• Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.91)