Research map / Video / Video generation
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self Forcing, a post-training paradigm that unrolls autoregressive self-generated rollouts during training of a few-step video diffusion model and supervises the full video with distribution-matching losses (DMD, SiD, GAN), plus a rolling KV cache for long-video extrapolation.
Problem
Exposure bias in autoregressive video diffusion: models trained with teacher forcing or diffusion forcing on ground-truth or noisy context accumulate errors at inference, degrading long-video quality and limiting real-time streaming generation.
Approach
Perform autoregressive self-rollout with KV caching during training so each frame/chunk is conditioned on the model's own outputs, using a few-step diffusion backbone, stochastic gradient truncation, and a video-level distribution-matching loss; a rolling KV cache with local-attention training enables artifact-free extrapolation.
Key results
- Chunk-wise Self Forcing (DMD) reaches VBench total 84.31 at 17.0 FPS and 0.69 s latency on a single H100, the highest total among compared models (Wan2.1-1.3B 84.26 at 0.78 FPS).
- Frame-wise Self Forcing achieves 0.45 s latency at 8.9 FPS with VBench total 84.26, while CausVid reaches 81.20 total under chunk-wise setting.
- Self Forcing with DMD converges in about 1.5 hours on 64 H100 GPUs and outperforms TF and DF baselines under equal wall-clock training budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Self Forcing rather than DF + DMD | Chunk-wise AR video diffusion · VBench | Total Score: 82.76 -> 84.31 | Helped |
| Train with Self Forcing rather than TF + DMD | Chunk-wise AR video diffusion · VBench | Total Score: 82.32 -> 84.31 | Helped |
| Train with Self Forcing rather than DF + DMD | Frame-wise AR video diffusion · VBench | Total Score: 80.56 -> 84.26 | Helped |
| Train with Self Forcing rather than TF + DMD | Frame-wise AR video diffusion · VBench | Total Score: 78.12 -> 84.26 | Helped |
| Replace DMD with SiD as the distribution-matching objective (ablation) | Chunk-wise Self Forcing · VBench | Quality Score: 85.07 -> 85.52 | Helped |
| Replace DMD with SiD as the distribution-matching objective (ablation) | Chunk-wise Self Forcing · VBench | Semantic Score: 81.28 -> 78.24 | Hurt |
| Replace DMD with GAN as the distribution-matching objective (ablation) | Chunk-wise Self Forcing · VBench | Total Score: 84.31 -> 83.88 | Hurt |
| Replace DMD with SiD as the distribution-matching objective (ablation) | Frame-wise Self Forcing · VBench | Total Score: 84.26 -> 83.54 | Hurt |
| Replace DMD with GAN as the distribution-matching objective (ablation) | Frame-wise Self Forcing · VBench | Total Score: 84.26 -> 83.27 | Hurt |
| Restrict attention so the last chunk cannot attend to the first chunk during training (ablation) | Rolling KV cache video extrapolation · Video extrapolation beyond training context length | Visual artifacts | Helped |
| Use a rolling KV cache instead of recomputing KV when shifting the sliding window (ablation) | Video extrapolation · 10-second videos | Throughput (FPS): 4.6 FPS -> 16.1 FPS | Helped |
Surprising findings
- Self Forcing training is not slower per iteration than teacher forcing or diffusion forcing; the sequential rollout is offset by avoiding attention masks and using FlashAttention-3.
- Naive rolling KV cache causes severe flickering artifacts because the model has always seen the first latent frame as an image latent; restricting training attention to avoid the first chunk fixes this.
Limitations
- Quality degrades when generating videos substantially longer than the training context length.
- Gradient truncation for memory efficiency may limit learning of long-range dependencies.
- Requires a strong pretrained teacher/real score network (Wan2.1-14B for DMD), adding training cost and dependence on the base model.
Experimental setup
Datasets: VidProM MovieGenBench
Benchmarks: VBench MovieGenBench user study
Metrics: VBench total score VBench quality score VBench semantic score throughput (FPS) first-frame latency (s) user preference rate
Baselines: Wan2.1-1.3B LTX-Video SkyReels-V2 MAGI-1 CausVid NOVA Pyramid Flow Teacher Forcing Diffusion Forcing
Builds on: Diffusion Forcing Teacher Forcing CausVid Distribution Matching Distillation Wan2.1
Closest papers by meaning
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.91)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.90)
- RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO (NeurIPS 2026 · similarity 0.90)
- Taming Teacher Forcing for Masked Autoregressive Video Generation (CVPR 2025 · similarity 0.89)
- Video-Mirai: Autoregressive Video Diffusion Models Need Foresight (NeurIPS 2026 · similarity 0.89)
- Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.