Research map / Video / Video generation
Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ca2-VDM makes autoregressive video diffusion efficient by using causal temporal attention so clean conditional-frame KV caches can be precomputed once, shared across all denoising steps, and kept in a queue with cyclic temporal positional embeddings.
Problem
Autoregressive video diffusion models recompute overlapping conditional frames at every denoising step and autoregression step, so cost grows quadratically with extended long-term context, and bidirectional temporal attention prevents KV caching.
Approach
Replace bidirectional temporal attention with causal temporal attention and use prefix-enhanced spatial attention, so clean prefix KV caches are independent of the noisy frames and timestep. Using tEmb(0) for clean prefix frames lets one cache be shared across all denoising steps, stored in a queue with cyclic TPEs.
Key results
- Zero-shot FVD on MSR-VTT is 181 for Ca2-VDM, tied with SEINE and better than the other listed T2V and image-conditioned baselines.
- Ca2-VDM's per-chunk FVD on MSR-VTT at AR step 2 is 160.6 versus 182.9 for OS-Fix, and it stays comparable to OS-Fix at later steps.
- GPU memory for the KV-cache with 50 denoising steps is 0.86 GB for Ca2-VDM (w/ PE) versus 17.70 GB for Live2diff, because cache sharing removes the per-timestep cache.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use extendable long-term conditional frames rather than fixed-length conditions | Autoregressive video generation · MSR-VTT test, later AR chunks | FVD: 182.9 -> 160.6; 210.6 -> 206.5; 284.3 -> 281.3; 315.1 -> 304.7 | Helped |
| Replace bidirectional generation with causal generation and cache sharing | Ca2-VDM · Autoregressive video generation | generation time | Helped |
| Share the conditional-frame KV-cache across denoising steps | Ca2-VDM · Video generation inference | GPU memory usage | Helped |
| Increase Pmax from 25 to 41 (ablation) | Ca2-VDM · SkyTimelapse test, chunks 1–3 | FVD: 274.8 -> 187.3; 244.5 -> 209.3; 275.1 -> 263.2 | Helped |
| Add prefix-enhancement to spatial attention (Pmax 25) (ablation) | Ca2-VDM · SkyTimelapse test, chunks 1–3 | FVD: 274.8 -> 257.4; 244.5 -> 216.5; 275.1 -> 238.5 | Helped |
| Add prefix-enhancement to spatial attention (Pmax 41) (ablation) | Ca2-VDM · SkyTimelapse test, chunks 1–3 | FVD: 187.3 -> 185.0; 209.3 -> 202.9; 263.2 -> 240.5 | Mixed |
| Use prefix-enhancement with Pmax 41 instead of 25 (ablation) | Ca2-VDM · SkyTimelapse test, chunks 1–3 | FVD: 257.4 -> 185.0; 216.5 -> 202.9; 238.5 -> 240.5 | Helped |
Surprising findings
- Ca2-VDM's cumulative time grows linearly with the autoregression step, while OS-Ext's grows quadratically before P_max is reached.
Limitations
- Causal temporal attention is fine-tuned from bidirectionally pretrained weights, which may be suboptimal; causal pretraining was not done due to compute.
- Extendable conditions and cyclic TPEs lengthen training convergence, and training time grows with P_max.
- Quality degradation in long-term generation persists and is not fully addressed.
Experimental setup
Datasets: InternVid SkyTimelapse MSR-VTT UCF-101
Benchmarks: MSR-VTT UCF101 SkyTimelapse VBench
Metrics: FVD time cost FLOPs GPU memory VBench aesthetic quality imaging quality motion smoothness temporal flickering
Baselines: OS-Fix OS-Ext GenLV StreamT2V Live2diff ModelScope VideoComposer Video-LDM PYoCo Make-A-Video AnimateAnything PixelDance
Builds on: Open-Sora PixArt-alpha DDPM improved DDPM ViT
Closest papers by meaning
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.90)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.90)
- Autoregressive Video Generation without Vector Quantization (ICLR 2025 · similarity 0.90)
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (NeurIPS 2025 · similarity 0.89)
- ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation (ICLR 2025 · similarity 0.89)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.