Research map / Video / Video generation
Real-Time Video Generation with Pyramid Attention Broadcast
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pyramid Attention Broadcast (PAB), a training-free method that reuses attention outputs across diffusion steps with pyramid-style broadcast ranges per attention type, plus broadcast sequence parallelism, to speed up DiT-based video generation.
Problem
Video DiTs are slow because spatial, temporal and cross attention are costly, and prior training-free caching methods were designed for U-Nets or image DiTs and do not fit video DiT attention structure.
Approach
Attention output differences across diffusion steps follow a U-shape, so attention outputs are broadcast to subsequent steps in the stable middle segment, with broadcast range set per attention type (spatial, temporal, cross) according to its variance; broadcast sequence parallel removes communication for broadcast temporal attention.
Key results
- PAB achieves up to 10.5x speedup on 8 H100 GPUs with broadcast sequence parallel, reaching real-time generation (up to 35.6 FPS) for 480p video.
- On a single GPU, PAB247 variants reach 1.34x-1.58x speedup on Open-Sora, Open-Sora-Plan and Latte while keeping VBench near original (e.g., Open-Sora 78.51 vs 79.22 original for PAB246).
- Compared with Δ-DiT, PAB yields much higher PSNR/SSIM and lower LPIPS; Δ-DiT drops Latte VBench to 52.00 from 77.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PAB_246 attention broadcasting instead of the original inference | Open-Sora · Open-Sora (2s, 480p) | latency (s): 26.54 -> 19.87 | Helped |
| Use PAB_246 attention broadcasting instead of the original inference | Open-Sora-Plan · Open-Sora-Plan (2.7s, 512x512) | latency (s): 46.49 -> 33.83 | Helped |
| Use PAB_235 attention broadcasting instead of the original inference | Latte · Latte (2s, 512x512) | latency (s): 11.18 -> 8.91 | Helped |
| Use PAB_246 attention broadcasting instead of T-GATE | Open-Sora · Open-Sora (2s, 480p) | VBench: 77.61 -> 78.51 | Mixed |
| Use PAB_246 attention broadcasting instead of T-GATE | Open-Sora-Plan · Open-Sora-Plan (2.7s, 512x512) | VBench: 80.15 -> 80.30 | Mixed |
| Use PAB_235 attention broadcasting instead of T-GATE | Latte · Latte (2s, 512x512) | VBench: 75.42 -> 76.32 | Mixed |
| Increase attention broadcast ranges from 2/4/6 to 3/5/7 (ablation) | PAB on Open-Sora · Open-Sora (16s, 480p) | total: 73.98 -> 72.58 | Hurt |
| Increase attention broadcast ranges from 2/4/6 to 5/7/9 (ablation) | PAB on Open-Sora · Open-Sora (16s, 480p) | total: 73.98 -> 72.15 | Hurt |
| Add attention broadcast computation speedup to DSP (ablation) | Open-Sora · Open-Sora (8s, 480p; 8 GPUs) | latency (s): 11.53 -> 9.29 | Helped |
| Add attention broadcast communication speedup to PAB computation speedup (ablation) | Open-Sora · Open-Sora (8s, 480p; 8 GPUs) | latency (s): 9.29 -> 8.85 | Helped |
| Disable spatial attention broadcasting (ablation) | PAB on Open-Sora · Open-Sora (2s, 480p) | VBench | No clear effect |
| Disable temporal attention broadcasting (ablation) | PAB on Open-Sora · Open-Sora (2s, 480p) | VBench | No clear effect |
Surprising findings
- Broadcasting attention outputs works better than broadcasting attention scores, which also prevents use of FlashAttention.
- Attention computation itself is a small share of attention module time; layernorm and positional embedding dominate runtime.
Limitations
- Performance may vary with input complexity, especially dynamic scenes; fixed broadcast strategy may not suit all video types.
- Only attention redundancy is reduced; other model parts are not addressed.
- Aggressive broadcast ranges degrade VBench on some models (e.g., Open-Sora-Plan PAB579 at 71.81).
Experimental setup
Datasets: VBench prompts WebVid (1000-video subset)
Benchmarks: VBench WebVid
Metrics: VBench score PSNR LPIPS SSIM FLOPs latency speedup FPS
Baselines: Δ-DiT T-GATE
Builds on: T-GATE Δ-DiT DSP
Closest papers by meaning
- PEEKABOO: Interactive Video Generation via Masked-Diffusion (CVPR 2024 · similarity 0.87)
- MonarchRT: Efficient Attention for Real-Time Video Generation (NeurIPS 2026 · similarity 0.87)
- MobileWan: Closing the Quality Gap for Mobile Video Diffusion (NeurIPS 2026 · similarity 0.86)
- Fast Video Generation with Sliding Tile Attention (ICML 2025 · similarity 0.86)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.86)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.