Research map / Architectures and efficiency / Attention and transformers

Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: Block-sparse attention kernels · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Training-free sparse attention for video DiTs that clusters Q/K tokens semantically with k-means, permutes them into contiguous blocks, and selects critical clusters by top-p budget with a custom dynamic-block-size kernel.

Problem

Sparse attention for video diffusion transformers loses generation quality at a given compute budget because position-based block identification is inaccurate and scattered critical tokens waste GPU computation on padding.

Approach

Cluster Q and K tokens by semantic similarity with k-means, permute tokens so each cluster is contiguous (output-equivalent), estimate cluster criticality from centroids, select clusters via top-p, and run a customized dynamic-block-size sparse attention kernel with a centroid cache to amortize k-means.

Key results

• On HunyuanVideo, SVG2 reaches 2.30x end-to-end speedup with PSNR 30.452 at 25.45% density, versus SVG's 1.91x with PSNR 29.157.
• On Wan2.1-I2V-14B, SVG2 reaches PSNR 26.562 at 31.28% density (1.58x), versus SVG's PSNR 24.059 at 30.25% density; the Turbo variant gives 1.84x speedup.
• The centroid cache reduces k-means latency by up to 76x, and the dynamic-block kernel achieves an average 1.48x computation reduction over FlashInfer.

Measured effects reported in the paper

Surprising findings

• Using the same clustering for Q and K (shared permutation or shared QK embedding) degrades PSNR substantially, with independent Q and K clustering being necessary (average ARI of 0.345 between Q and K clusters).

Limitations

• Evaluation limited to DiT-based video generation; extension to other attention mechanisms is not discussed or evaluated.
• Quality metrics (PSNR/SSIM/LPIPS) measure similarity to dense attention rather than absolute quality; VBench differences are small.
• Wan 2.1 shows lower PSNR than HunyuanVideo across all methods, attributed to numerical sensitivity, and XAttention was not evaluated on Wan 2.1.

Experimental setup

Datasets: VBench prompts Penguin Benchmark prompts

Benchmarks: VBench

Metrics: PSNR SSIM LPIPS VBench density FLOPs speedup attention recall

Baselines: SpargeAttention Sparse VideoGen (SVG) XAttention FlashInfer

Builds on: Sparse VideoGen SpargeAttention FlashInfer FlashAttention-3 MMInference

Closest papers by meaning

• Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.93)
• iCATS: Fast Video Generation via Interaction-Aware Sparse Attention and Timestep-Adaptive Sparsity (NeurIPS 2026 · similarity 0.89)
• Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.89)
• MonarchRT: Efficient Attention for Real-Time Video Generation (NeurIPS 2026 · similarity 0.88)
• HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion (NeurIPS 2026 · similarity 0.88)
• Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.88)