Research map / Video / Video generation
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a many-step bidirectional video diffusion transformer into a 4-step causal (block-wise autoregressive) generator using DMD with asymmetric teacher-student attention and ODE-based initialization, enabling streaming video generation at 9.4 FPS.
Problem
Video diffusion transformers use bidirectional attention, so generating any frame requires the full sequence, causing high latency and preventing interactive/streaming generation; existing autoregressive video models suffer from error accumulation and are slow.
Approach
Adapt a pretrained bidirectional DiT into a block-wise causal autoregressive student and distill it with distribution matching distillation using a bidirectional teacher (asymmetric distillation), with ODE-trajectory regression initialization to stabilize training; KV caching enables streaming inference.
Key results
- Achieves 9.4 FPS throughput and 1.3 s initial latency on 10-second 120-frame video on H100, versus 219.2 s for bidirectional teacher.
- Reaches VBench-Long total score of 84.27, reported as first place among officially evaluated video generation models.
- Text-to-short-video temporal quality 94.7, frame quality 64.4, text alignment 30.1, outperforming CogVideoX-5B, MovieGen, and Pyramid Flow on the VBench subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill a bidirectional video diffusion teacher into a 4-step causal generator | CausVid · Text-to-video, short (VBench) | temporal quality: 91.5 -> 94.7 | Helped |
| Distill a bidirectional video diffusion teacher into a 4-step causal generator | CausVid · Text-to-video, short (VBench) | frame quality: 61.1 -> 64.4 | Helped |
| Distill a bidirectional video diffusion teacher into a 4-step causal generator | CausVid · Text-to-video, short (VBench) | text alignment: 29.1 -> 30.1 | Helped |
| Use asymmetric DMD with a bidirectional teacher instead of a causal teacher (ablation) | CausVid · 10-second text-to-video | temporal quality: 91.9 -> 94.7 | Helped |
| Use asymmetric DMD with a bidirectional teacher instead of a causal teacher (ablation) | CausVid · 10-second text-to-video | frame quality: 61.7 -> 64.4 | Helped |
| Use asymmetric DMD with a bidirectional teacher instead of a causal teacher (ablation) | CausVid · 10-second text-to-video | text alignment: 28.2 -> 30.1 | Helped |
| Initialize the student by fitting teacher ODE trajectory pairs (ablation) | CausVid · 10-second text-to-video | temporal quality: 93.4 -> 94.7 | Helped |
| Initialize the student by fitting teacher ODE trajectory pairs (ablation) | CausVid · 10-second text-to-video | frame quality: 60.6 -> 64.4 | Helped |
| Initialize the student by fitting teacher ODE trajectory pairs (ablation) | CausVid · 10-second text-to-video | text alignment: 29.4 -> 30.1 | Helped |
| Replace the bidirectional teacher with no teacher (ablation) | CausVid · 10-second text-to-video | frame quality: 64.4 -> 48.1 | Hurt |
| Fine-tune the bidirectional DiT into a causal model without few-step distillation (ablation) | Bidirectional DiT · 10-second text-to-video | temporal quality: 94.6 -> 92.4 | Hurt |
| Fine-tune the bidirectional DiT into a causal model without few-step distillation (ablation) | Bidirectional DiT · 10-second text-to-video | frame quality: 62.7 -> 60.1 | Hurt |
Surprising findings
- Naively distilling or fine-tuning a causal teacher yields worse students; a bidirectional teacher with causal student avoids error accumulation seen in causal teachers.
- The student improves frame-level quality over the bidirectional teacher but is worse on temporal flickering and output diversity.
Limitations
- Trained on internal proprietary video data, limiting reproducibility of training data.
- Student underperforms teacher on temporal flickering and output diversity.
- Image-to-video is zero-shot without instruction fine-tuning, which the authors note could be improved.
Experimental setup
Datasets: internal video dataset (~400K single-shot videos) MovieGen benchmark prompts VBench prompts
Benchmarks: VBench VBench-Long VBench-I2V MovieGen benchmark prompts
Metrics: temporal quality frame quality text alignment total score latency (s) throughput (FPS) human preference win rate
Baselines: CogVideoX-5B OpenSORA Pyramid Flow MovieGen Gen-L-Video FreeNoise StreamingT2V FIFO-Diffusion StreamV2V
Builds on: Distribution matching distillation (DMD) DMD2 Diffusion Forcing CogVideoX FlexAttention
Closest papers by meaning
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.93)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.92)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.91)
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (NeurIPS 2025 · similarity 0.90)
- ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation (ICLR 2025 · similarity 0.90)
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.