Research map / Image generation / Fast diffusion sampling
FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FPSAttention, a training-aware co-design of FP8 quantization and tile-wise structured sparse attention for Wan2.1 video DiTs, with a denoising step-aware schedule, achieving up to 4.96x end-to-end speedup on VBench.
Problem
Video diffusion transformers are slow because of many denoising steps and quadratic 3D attention; combining training-free FP8 quantization with sparse attention causes large quality degradation, and existing methods lack training-aware joint optimization of the two.
Approach
Unify FP8 quantization and block sparsity at a shared 3D tile granularity, vary quantization tile size and sparsity window across denoising steps (early/mid/late), and train the model with these joint approximations so it adapts to the errors, with a FlexAttention/Triton kernel for wall-clock speedups.
Key results
- On Wan2.1-14B at 720p, FPSAttention reaches 4.96x end-to-end speedup versus BF16 baseline, versus 3.60x for STA.
- On Wan2.1-1.3B (480p), Ours Quant+Sparse achieves PSNR 21.35 and 2.45x speedup, versus 1.89x for STA.
- On Wan2.1-14B, Ours Quant+Sparse reports PSNR 25.74 vs 22.66 for STA; kernel speedup up to 7.09x on H20 in the abstract/intro table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly train FP8 quantization and structured sparsity with denoising-step-aware adaptation (ablation) | Wan2.1-1.3B · VBench | Total Score: 0.8019 -> 0.8160; +1.8% | Helped |
| apply FP8 quantization and sparse attention without joint optimization or denoising step-aware adaptation (ablation) | Wan2.1-1.3B · VBench | Total Score: 0.8019 -> 0.6325; -21.1% | Hurt |
| add structured sparsity to quantization (ablation) | Wan2.1-1.3B Ours Quant · VBench | PSNR: 20.99712 -> 21.35417 | Helped |
| increase quantization and sparsity tile size from (3,4,4) to (24,32,32) (ablation) | FPSAttention · Wan2.1-1.3B | PSNR: 19.87452 -> 20.99712 | Helped |
| use hardware-friendly tile size (6,8,8) instead of unbalanced tile size (3,4,4) (ablation) | FPSAttention · Wan2.1-1.3B | PSNR: 19.87452 -> 20.12358 | Helped |
| use sparsity window (6,6,1) instead of (6,6,6) (ablation) | FPSAttention · Wan2.1-1.3B | Kernel Speedup: 1.69x -> 5.16x | Helped |
| increase sparsity with window (3,3,1) rather than (6,6,1) (ablation) | FPSAttention · Wan2.1-1.3B | PSNR: 20.45621 -> 19.23465 | Hurt |
| use joint FP8 quantization and structured sparsity | Wan2.1-1.3B · training after 2,000 steps | Training loss difference: <2% | No clear effect |
| use joint FP8 quantization and structured sparsity | Wan2.1-1.3B · training initially | Training loss: ~15% | Hurt |
| joint FP8 quantization and structured sparsity instead of FP8 quantization alone (ablation) | Wan2.1-14B · 480p video generation | Latency: 144s -> 423s | Helped |
| joint FP8 quantization and structured sparsity instead of STA | Wan2.1-14B · 720p video generation | Speedup: 3.60x -> 4.96x | Helped |
Surprising findings
- Training-free naive FP8 plus sparsity drops the VBench total score by 21.1% on Wan 1.3B, while the jointly trained version slightly exceeds the baseline (0.8160 vs 0.8019).
- Joint training yields slightly higher VBench scores than the baseline, which the authors attribute possibly to locality inductive bias.
Limitations
- Performs best on FP8-supporting hardware (Hopper); benefits reduced on older GPUs.
- Requires quantization-aware training with extra training cost and several hyperparameters (transition points, granularities, window sizes).
- Validated only on the Wan2.1 architecture.
Experimental setup
Datasets: curated high-quality internal video dataset VBench
Benchmarks: VBench PSNR SSIM LPIPS
Metrics: VBench total score imaging quality subject consistency PSNR SSIM LPIPS FLOPS latency speedup
Baselines: SageAttention SpargeAttn SparseVideoGen Sliding Tile Attention
Builds on: Sliding Tile Attention SageAttention FlexAttention FlashAttention
Closest papers by meaning
- DSAQuant: Denoising-Stage-Aligned Quantization-Aware Training for Video Generation (NeurIPS 2026 · similarity 0.89)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.88)
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.88)
- ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation (ICLR 2025 · similarity 0.88)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.88)
- $\text{S}^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.