Research map / Video / Video generation
Generative Pre-trained Autoregressive Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GPDiT, a frame-wise autoregressive diffusion transformer over continuous video latents with a lightweight causal attention variant and a parameter-free rotation-based time conditioning, evaluated on video generation, linear-probe representation, and few-shot video-task adaptation.
Problem
Diffusion video models lack causal structure and struggle with temporal consistency and long-range generation, while discrete-token autoregressive video models are lossy; the paper targets efficient, causal, long-range video modeling in continuous latent space.
Approach
Autoregressively predict future latent frames with a diffusion loss under frame-wise causal attention, drop attention between clean frames to save compute (lightweight causal attention), and replace adaLN-Zero with a parameter-free rotation of data and noise components for timestep conditioning.
Key results
- GPDiT-H-LONG achieves FID 7.4 and FVD 64 zero-shot on MSR-VTT, and FID 7.9, FVD 218, IS 66.6 on UCF-101.
- GPDiT-B-OF2 (80M params) trained on UCF-101 reaches FVD 214, outperforming FAR (194 is better; GPDiT is second-best among listed Frame-AR methods behind FAR at 194).
- On ActivityNet-QA with a frozen first-18-layer GPDiT-H feature extractor in Video-ChatGPT, accuracy is 28.2% vs 35.2% for original CLIP-based Video-ChatGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train GPDiT-H-LONG on 24M video data with variable-length video training | GPDiT-H · MSRVTT | FVD: 68 -> 64 | Helped |
| Train GPDiT-H-LONG on 24M video data with variable-length video training | GPDiT-H · UCF-101 | FID: 14.8 -> 7.9 | Helped |
| Train GPDiT-H-LONG on 24M video data with variable-length video training | GPDiT-H · UCF-101 | FVD: 243 -> 218 | Helped |
| Train GPDiT-H-LONG on 24M video data with variable-length video training | GPDiT-H · UCF-101 | IS: 66.5 -> 66.6 | Helped |
| Allow interactions between clean context frames (OF2 instead of OF) (ablation) | GPDiT-B · UCF-101 linear probing | top-1 accuracy | Helped |
| Use lightweight causal attention that eliminates attention computation between clean frames (ablation) | GPDiT-B · Video generation training convergence | training convergence speed | Helped |
| Replace adaLN-Zero with parameter-free rotation-based time conditioning (ablation) | GPDiT · Video generation | generation performance | No clear effect |
| Use GPDiT-H instead of Video-ChatGPT's CLIP vision encoder as the feature extractor | Video-ChatGPT · ActivityNet-QA | accuracy: 35.2 -> 28.2 | Hurt |
Surprising findings
- Pre-training on a video-only setting with a frozen GPDiT encoder achieves only moderately lower ActivityNet-QA accuracy (28.2% vs 35.2%) despite never being trained on language.
- Classification accuracy in the zero-shot GPDiT-H model peaks at a deeper 2/3 layer, unlike the GPDiT-B model where it peaks at shallow layers.
Limitations
- Authors state limited compute prevented scaling to larger configurations.
- Currently limited to the video modality; multimodal extension is future work.
- Few-shot and in-context learning claims are qualitative; larger models' ICL potential untested.
Experimental setup
Datasets: UCF-101 MSR-VTT LAION-Aesthetic MovieGenBench ActivityNet-QA
Benchmarks: MSR-VTT UCF-101 ActivityNet-QA
Metrics: FID FVD IS top-1 accuracy (linear probing) accuracy score
Baselines: MagicVideo LVDM ModelScope PixelDance DreamVideo SnapVideo CogVideo InternVid Video-LDM Make-A-Video ExtDM-K2 OmniTokenizer
Builds on: DiT SiT Diffusion Forcing adaLN-Zero REPA
Closest papers by meaning
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.93)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.90)
- Autoregressive Video Generation without Vector Quantization (ICLR 2025 · similarity 0.90)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.89)
- Video-Mirai: Autoregressive Video Diffusion Models Need Foresight (NeurIPS 2026 · similarity 0.89)
- Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.