Research map / Video / Video generation
VORTA: Efficient Video Diffusion via Routing Sparse Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VORTA, a training-light framework that routes each attention block among full, sliding-tile, and bucketed core-set sparse attention using timestep-conditioned gates, achieving 1.76x speedup on HunyuanVideo with lossless VBench quality and up to 14.41x when combined with step distillation.
Problem
Video diffusion transformers are expensive because full 3D attention is quadratic in the very long video token sequence, and existing sparse methods handle long-range attention poorly or add profiling overhead.
Approach
Combine a local sliding-tile attention and a new linear-cost bucketed core-set attention (token pooling that drops redundant tokens) with full attention, and learn a timestep-conditioned softmax router that selects one branch per block, trained only on the router with a distillation loss against the frozen model.
Key results
- VORTA achieves 1.76x end-to-end speedup on HunyuanVideo with VBench 82.59 vs 82.26 for the dense baseline.
- VORTA combined with PAB reaches 2.35x speedup and combined with PCD reaches 14.41x speedup on HunyuanVideo.
- On Wan 2.1 (14B), VORTA gives 1.52x speedup with VBench 82.85 vs 82.36 for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sparse attention routing (VORTA) over full/sliding/core-set attention | HunyuanVideo · VBench 720p 5s | VBench: 82.59 vs 82.26 baseline | Helped |
| sparse attention routing (VORTA) | HunyuanVideo · VBench 720p 5s | latency speedup: 1.76x speedup | Helped |
| VORTA combined with PAB caching | HunyuanVideo · VBench 720p 5s | latency speedup: 2.35x vs 1.76x VORTA alone | Helped |
| VORTA combined with PCD step distillation | HunyuanVideo · VBench 720p 5s | latency speedup: 14.41x vs 8.29x PCD alone, with VBench 81.49 vs 81.17 | Helped |
| sparse attention routing (VORTA) | Wan 2.1 (14B) · VBench 720p 5s | VBench: 82.85 vs 82.36 | Helped |
| removing the full attention branch from the router (ablation) | VORTA (Wan 2.1 1.3B ablation) · 480p ablation | VBench: about 4-point drop in VBench | Hurt |
| removing the sliding attention branch (ablation) | VORTA router (Wan 2.1 1.3B ablation) · 480p ablation | latency: over 10% runtime increase | Hurt |
| removing the core-set attention branch (ablation) | VORTA router (Wan 2.1 1.3B ablation) · 480p ablation | latency: latency increased by 14% | Hurt |
| removing timestep embedding from router input (ablation) | VORTA router (Wan 2.1 1.3B ablation) · 480p ablation | latency: slower by 12% | Hurt |
| average pooling instead of bucketed core-set selection (BCS) (ablation) | core-set attention branch · Wan 2.1 1.3B 480p ablation | VBench: performance drops significantly; average pooling only slightly lower latency | Hurt |
| VORTA routing on HunyuanVideo vs. STA predefined sparse pattern | HunyuanVideo · VBench 720p 5s | latency speedup: 1.76x vs 1.54x for STA | Helped |
| PCD step distillation (6 steps) alone | HunyuanVideo · VBench 720p 5s | VBench: 81.17 vs 82.26 baseline | Hurt |
Surprising findings
- VORTA slightly outperforms the dense pretrained models on VBench despite using less computation.
- Full attention is selected in fewer than 0.2% of cases, yet removing it drops VBench by about 4 points.
Limitations
- Gains are limited for short sequences such as image or low-resolution video, where attention is not dominant.
- Only bidirectional generation is supported; autoregressive paradigms are not directly handled.
- VORTA inherits failure modes of the pretrained VDiT and can sometimes produce lower quality outputs.
Experimental setup
Datasets: Mixkit
Benchmarks: VBench
Metrics: VBench score VBench quality VBench semantic LPIPS latency speedup peak memory
Baselines: ARnR STA PAB PCD SVG
Builds on: sliding tile attention FlashAttention conditional flow matching PAB PCD
Closest papers by meaning
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.91)
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.89)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.89)
- PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers (NeurIPS 2026 · similarity 0.89)
- AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration (ICML 2025 · similarity 0.88)
- HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.