Research map / Video / Video generation
Autoregressive Video Generation without Vector Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NOVA, a non-quantized autoregressive model that predicts video frames temporally (causal, GPT-style) and spatial token sets in random order with per-token diffusion loss, unifying text-to-image and text-to-video generation.
Problem
Vector-quantized autoregressive visual generators are costly at high fidelity and resolution, while video diffusion models use fixed-length joint modeling and lack in-context/autoregressive flexibility for variable-length and multi-task generation.
Approach
Reformulates video generation as causal frame-by-frame temporal prediction combined with bidirectional, random-order set-by-set spatial prediction over continuous (non-quantized) VAE latents, with a per-token diffusion loss, a scaling-and-shift layer conditioned on the BOV-attended anchor frame, and post-norm before residual for stable training.
Key results
- NOVA (0.6B) reaches VBench total 78.48 (80.12 with rewriter) vs Emu3 (8B) 80.96 and CogVideo (9B) 67.01.
- NOVA (0.6B, 1024px) achieves GenEval overall 0.69 (0.75 with rewriter at 512px) and DPG-Bench 82.25 with ~127-342 A100 GPU days.
- Ablation: without temporal AR modeling, VBench total drops (75.38 vs 75.84) and dynamic degree falls from 23.27 to 11.38; inference time rises from 12s to 39s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal autoregressive modeling (TAM) (ablation) | NOVA · VBench | Total Score: 75.38 vs. 75.84 | Helped |
| Add temporal autoregressive modeling (TAM) (ablation) | NOVA · VBench | Dynamic Degree: 11.38 vs. 23.27 | Helped |
| Add temporal autoregressive modeling (TAM) (ablation) | NOVA · Video inference | inference time: 39s vs. 12s | Helped |
| Use rewritten prompts | NOVA · GenEval | Overall: 0.66 -> 0.75 | Helped |
| Use rewritten prompts | NOVA · VBench | Total Score: 78.48 -> 80.12 | Helped |
| Replace temporal autoregressive modeling with spatial-only autoregressive generation (ablation) | NOVA · Text-to-video generation | subject movement | Hurt |
| Remove the Scaling and Shift Layer (ablation) | NOVA · Text-to-video generation | training-loss drift | Hurt |
| Reduce the Scaling and Shift Layer MLP inner rank (ablation) | NOVA · Text-to-video generation | motion modeling | Mixed |
| Use pre-normalization instead of post-normalization before residual addition (ablation) | NOVA · Model training | training stability | Hurt |
| Use stochastic depth or residual dropout on the residual branch (ablation) | NOVA · Model training | training stability | No clear effect |
Surprising findings
- Without temporal autoregressive modeling, dynamic degree drops sharply (23.27 to 11.38) even though total VBench score changes little.
- Pre-normalization training suffered numerical overflow and variance instability, while post-norm before residual fixed it, and stochastic depth and residual dropout were less effective.
Limitations
- Current model generates fixed 33-frame videos and extends length via pre-filling; extrapolation quality degrades beyond about 3x training length.
- Text rendering and DPG-Bench gains do not improve with scale, attributed to reliance on web datasets such as LAION and DataComp.
- Spatial AR layers remain the inference bottleneck (11.97s of 12s).
Experimental setup
Datasets: DataComp COYO Unsplash JourneyDB LAION Panda-70M Pexels
Benchmarks: GenEval T2I-CompBench DPG-Bench VBench
Metrics: GenEval overall score T2I-CompBench color/shape/texture DPG-Bench overall VBench total score latency (s) FPS LPIPS PSNR
Baselines: PixArt-α SD v1.5 SD v2.1 SDXL DALL-E 2 DALL-E 3 SD3 LlamaGen Emu3 CogVideo Gen-2 Kling
Builds on: MAR Emu3 VAR LlamaGen DiT
Closest papers by meaning
- VideoMAR: Autoregressive Video Generation with Continuous Tokens (NeurIPS 2025 · similarity 0.90)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
- Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing (ICML 2025 · similarity 0.90)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.89)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.89)
- InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.