Research map / Video / Video generation
LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LARP, a video tokenizer that uses learned holistic queries and a co-trained lightweight AR prior to produce AR-friendly discrete tokens, reaching FVD 57 on UCF101 class-conditional generation.
Problem
Patchwise video tokenizers tie discrete tokens to local patches and require a manually chosen flattening order for autoregressive (AR) generation, and their latent spaces are not optimized for AR modeling.
Approach
Replace patch-to-token correspondence with learned holistic queries concatenated to patch embeddings, quantize with stochastic VQ, and co-train a discarded-at-inference continuous AR transformer prior whose NLL gradients shape the latent space and implicitly define a token order.
Key results
- LARP-L-Long with a 632M AR generator reaches gFVD 57 on UCF-101, claimed state of the art among published video generators.
- On K600 frame prediction, LARP-L-Long (632M generator) reaches gFVD 5.1, versus 9.9 for MAGVIT-MLM and 32.9 for OmniTokenizer.
- Ablation on LARP-B: removing the AR prior raises gFVD from 107 to 190 while slightly improving rFVD (31 to 23).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the AR prior model (ablation) | LARP-B · UCF-101 | gFVD: 190 -> 107 | Helped |
| Add the AR prior model (ablation) | LARP-B · UCF-101 | rFVD: 23 -> 31 | Hurt |
| Add scheduled sampling in the AR prior (ablation) | LARP-B · UCF-101 | gFVD: 142 -> 107 | Helped |
| Use stochastic rather than deterministic quantization (ablation) | LARP-B · UCF-101 | gFVD: 149 -> 107 | Helped |
| Use the default AR prior loss weight rather than α=0.03 (ablation) | LARP-B · UCF-101 | gFVD: 120 -> 107 | Helped |
| Use classifier-free guidance (ablation) | LARP-B · UCF-101 | gFVD: 121 -> 107 | Helped |
| Train LARP-L for more epochs | LARP-L · UCF-101 class-conditional generation | gFVD: 107 -> 102 | Helped |
| Use a larger AR generator | LARP-L-Long · UCF-101 class-conditional generation | gFVD: 102 -> 57 | Helped |
Surprising findings
- Removing the AR prior gives the best reconstruction (rFVD 23) but the worst generation (gFVD 190), showing reconstruction quality and generation quality can diverge.
- gFVD saturates when scaling the tokenizer from LARP-B to LARP-L while rFVD keeps improving.
Limitations
- Performs best on fixed-resolution videos due to positional encoding constraints.
- Artifacts appear in videos with particularly complex scenes.
- Evaluated mainly on 128x128, 16-frame clips from UCF-101 and K600.
Experimental setup
Datasets: UCF-101 Kinetics-600
Benchmarks: UCF-101 class-conditional video generation Kinetics-600 frame prediction
Metrics: FVD rFVD gFVD PSNR LPIPS
Baselines: OmniTokenizer MAGVIT-v2-AR MAGVIT-AR TATS CogVideo VideoFusion HPDM MAGVIT-MLM MAGVIT-v2-MLM
Builds on: VQGAN ViT-VQGAN MAGVIT-v2 Perceiver-style learned queries GPT-2 Llama
Closest papers by meaning
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.91)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.90)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.89)
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.88)
- Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction (CVPR 2025 · similarity 0.88)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.