Research map / Video / Video generation
Fast Video Generation with Sliding Tile Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sliding Tile Attention (STA), a tile-wise sliding-window attention kernel for video DiTs that removes mixed blocks, giving near-proportional speedups on HunyuanVideo with training-free or lightly finetuned variants.
Problem
Full 3D attention in video diffusion transformers dominates inference cost (about 800 of 945 seconds for a 5s 720P HunyuanVideo clip), but existing 2D/3D sliding window attention implementations do not convert FLOP savings into wall-clock speedups.
Approach
Slide the attention window tile-by-tile (tiles equal to FlashAttention block size) rather than token-by-token, so every computed block is dense or empty with no mixed blocks; per-head window sizes are chosen by profiling, and optional finetuning with attention and output distillation losses allows sparser windows.
Key results
- STA on HunyuanVideo reduces 5s 720P latency from 945s (FA3) to 501s training-free (1.89x) with no quality loss, and to 268s with finetuning (3.53x, VBench 82.62% vs 82.71% for FA3).
- STA kernel achieves 58.79% MFU and 10.45x attention speedup over full attention at 91% sparsity on H100.
- In human preference on 200 MovieGen prompts, STA-tf-1.89x had an 83.0% tie rate with HunyuanVideo; STA-t-2.43x beat Delta-DiT-1.8x with a 70.0% win rate vs 11.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full attention with training-free STA at w=(30,40,40) | HunyuanVideo · VBench | Latency: 945s -> 527s | Helped |
| replace full attention with training-free STA at w=(30,40,40) | HunyuanVideo · VBench | VBench Total: 82.71% -> 82.46% | No clear effect |
| increase STA sparsity using w=(18,24,24) instead of w=(30,40,40), without training (ablation) | HunyuanVideo STA · VBench | VBench Total: 82.46% -> 80.58% | Hurt |
| increase STA sparsity using w=(18,24,24) instead of w=(30,40,40), without training (ablation) | HunyuanVideo STA · VBench | Latency: 527s -> 268s | Helped |
| fine-tune STA with w=(18,24,24) (ablation) | HunyuanVideo STA · VBench | VBench Total: 80.58% -> 82.62% | Helped |
| use the optimized ThunderKittens kernel instead of FlexAttention for STA w=(18,24,24) (ablation) | HunyuanVideo STA · attention kernel | Speedup: $7.30\times$ -> $10.45\times$ | Helped |
| replace Tiled NATTEN with STA using FlexAttention at similar sparsity | HunyuanVideo attention kernel · attention kernel, 720P 5s | Speedup: $1.27\times$ -> $7.30\times$ | Helped |
| use Δ-DiT caching instead of training-free STA | HunyuanVideo · 50 steps | SSIM: 87.67 vs. 72.86 | Hurt |
| use Δ-DiT caching instead of training-free STA | HunyuanVideo · 50 steps | CD-FVD: 66.12 vs. 122.74 | Hurt |
| replace full attention with Swin w=(48,64,64) | HunyuanVideo · VBench | VBench Total: 82.71% -> 79.00% | Hurt |
| fine-tune Swin attention at w=(30,40,40) (ablation) | HunyuanVideo · VBench | VBench Total: 77.53% -> 75.48% | Hurt |
| use STA w=(18,24,24) instead of w=(30,40,40), with fine-tuning (ablation) | HunyuanVideo STA · VBench | VBench Total: 83.00% -> 82.62% | Hurt |
Surprising findings
- Tiled NATTEN and CLEAR increase end-to-end latency despite sparsifying attention, due to kernel inefficiency.
- Swin attention's quality drops and finetuning does not recover it, whereas STA with finetuning recovers quality.
Limitations
- Quality evaluation relies heavily on human studies because automated metrics are judged unreliable; the training-free 1.89x setting still shows a 7-point lower win rate than HunyuanVideo.
- Training-free STA at high sparsity (91%) lowers VBench quality to 80.58% and needs finetuning to recover.
- Evaluation is mainly on HunyuanVideo; other video models (Wan 2.1) are only tested briefly in the appendix.
Experimental setup
Datasets: Mixkit prompts MovieGen Bench VBench prompts COCO-2014 validation
Benchmarks: VBench MovieGen Bench COCO-2014
Metrics: MFU latency TFLOPS speedup VBench score SSIM PSNR CD-FVD human win rate
Baselines: FlashAttention-2 FlashAttention-3 CLEAR NATTEN Tiled NATTEN Swin Delta-DiT
Builds on: FlashAttention-3 ThunderKittens NATTEN FlexAttention Delta-DiT
Closest papers by meaning
- Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.90)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.90)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.88)
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.87)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.87)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.