Research map / Architectures and efficiency / Quantization

PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

NeurIPS 2025 · Architectures and efficiency › Quantization · approach: Post-training quantization of ViTs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PAROAttention, which permutes token order per attention head so diverse visual attention patterns become block-wise, enabling 20-30% density sparse attention and INT8/INT4 quantized attention for CogVideoX, Wan 2.1, and Flux with 1.9-2.7x end-to-end speedup.

Problem

Quadratic attention cost in high-resolution image and long video diffusion transformers; existing sparse and quantized attention lose quality at low density (<50%) and low bitwidth (INT4/INT8) because visual attention patterns are dispersed and diagonal-like.

Approach

Reorder tokens per head (choosing among the 6 permutations of the F, H, W dimensions via an offline metric combining sparsity and incoherence) so that locally aggregated tokens become contiguous, turning multi-diagonal patterns into block-wise patterns that suit block-sparse masks and block-wise INT quantization of the PV product.

Key results

• At 20% density on CogVideoX, PAROAttn reaches PSNR 19.39 and VQA-style quality near FP16 (92.42 vs 92.53), while SpargeAttn at 50% density reaches PSNR 16.80.
• PAROAttn at 50% density achieves 1.73x speedup on A100 with PSNR 29.14, vs SpargeAttn 1.67x (PSNR 16.80) and SparseVideoGen 1.42x.
• Combined sparse plus INT4 PAROAttn reaches up to 9.28x speedup on RTX4090 (PSNR 24.34) versus 1.72x-2.56x for SageAttention baselines.

Measured effects reported in the paper

Surprising findings

• Post-reordering INT8 quantization of the attention map P gives higher fidelity than FP8, contrary to SageAttention2's choice of FP8 for PV.
• Reported speedups of 5.72x and 9.28x combine sparsity and quantization, far above the 1.7-2.7x of sparsity alone.

Limitations

• Permutations are limited to the 6 axis orderings of [F,H,W], a constrained subset of possible reorderings.
• Sparse metric is a simple block-sum threshold; more advanced metrics are unexplored.
• Static offline sparse masks require timestep-wise masks and calibration; per-head masks add memory overhead (about 1GB before prefetch).

Experimental setup

Datasets: CogVideo example prompts (calibration) VBench prompts (evaluation) COCO prompts (evaluation)

Benchmarks: VBench COCO

Metrics: CLIPSIM VQA FlowScore CLIPScore ImageReward PSNR SSIM CosSim FVD-FP16 FID-FP16

Baselines: DiTFastAttn MInference SpargeAttn SparseVideoGen RTN SageAttn SageAttnV2

Builds on: SageAttention SageAttention2 SpargeAttn FlashAttention incoherence-based quantization

Closest papers by meaning

• Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape (NeurIPS 2025 · similarity 0.90)
• Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.88)
• PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers (NeurIPS 2026 · similarity 0.88)
• Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.87)
• Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.87)
• MonarchRT: Efficient Attention for Real-Time Video Generation (NeurIPS 2026 · similarity 0.87)