Research map / Video / Video generation

PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

NeurIPS 2026 · Video › Video generation · approach: Efficient video diffusion models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PISA, a training-free piecewise sparse attention for diffusion transformers that computes critical KV blocks exactly and approximates the remaining blocks via block-wise Taylor expansion fused into online softmax.

Problem

Block sparse attention in video and image diffusion transformers discards non-critical key-value blocks, causing quality degradation at high sparsity, while linear/hybrid alternatives break pre-trained attention distributions and need retraining.

Approach

Replace the keep-or-drop rule with exact-or-approximate: exact computation for selected blocks and a hybrid-order (block-wise zeroth-order plus global first-order) Taylor approximation for unselected blocks, integrated into the softmax numerator and denominator so pre-trained weights are reused without training.

Key results

• On Wan2.1-14B at 87.5% sparsity, PISA reaches 1.91x speedup with PSNR 22.69 and LPIPS 0.124 versus SpargeAttn 1.85x and PSNR 21.47.
• On Hunyuan-13B 720P, PISA achieves 2.57x speedup, versus 2.51x for SpargeAttn and 2.54x for SVG2.
• On FLUX.1-dev at 85% sparsity, PISA gets PSNR 17.07 and LPIPS 0.241 versus SpargeAttn at 80% sparsity with PSNR 12.88 and LPIPS 0.296, with latency 6.87 s vs 7.47 s.

Surprising findings

• The zeroth-order-only variant achieves a higher speedup than the hybrid variant on FLUX.1-dev (1.21x vs 1.24x for hybrid, close), and the first-order-only variant is slower (0.96x), indicating the first-order term's memory cost matters.

Limitations

• Speedup on FLUX is modest (about 1.2x) because image generation has lower inherent sparsity.
• The Taylor truncation error is argued to be negligible but not bounded rigorously; the error theorem bounds only the global-mean replacement.
• Covariance-aware block selection is only used for image generation; the video results use the default routing.

Experimental setup

Benchmarks: VBench FID ImageReward HPSv2 MPS

Metrics: VBench S.C./I.Q./A.Q. SSIM PSNR LPIPS FID ImageReward HPSv2 MPS latency speedup

Baselines: SpargeAttn SVG2 FlashAttention-2 FlashAttention-3

Builds on: SpargeAttn FlashAttention SVG2

Closest papers by meaning

• Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.91)
• Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.90)
• Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.90)
• Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.90)
• MonarchRT: Efficient Attention for Real-Time Video Generation (NeurIPS 2026 · similarity 0.89)
• Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape (NeurIPS 2025 · similarity 0.89)