Research map / Video / Video generation
VideoPoet: A Large Language Model for Zero-Shot Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
VideoPoet is a decoder-only multimodal LLM that generates videos from discrete MAGVIT-v2 visual and SoundStream audio tokens plus T5 text embeddings, using multi-task pretraining and a token-space super-resolution transformer.
Problem
Video generation is dominated by diffusion models; the paper asks whether an LLM-style decoder-only transformer trained with a multitask multimodal objective can produce high-quality, motion-rich video and zero-shot video tasks.
Approach
Treat image, video, audio and text as tokens in one unified vocabulary and train a single autoregressive decoder-only transformer on a mixture of generative tasks (T2V, I2V, frame prediction, inpainting/outpainting, stylization, audio-video), followed by a token-space windowed-attention super-resolution transformer.
Key results
- Zero-shot MSR-VTT FVD of 213 for the pretrained VideoPoet versus 538 for Show-1 (best listed baseline), with CLIPSIM 0.3049 for pretrained and 0.3123 after task adaptation.
- On UCF-101 zero-shot, VideoPoet pretrain reaches FVD 355 and IS 38.44, beating Make-A-Video (367, 33.00) and Show-1 (394, 35.42).
- Human raters preferred VideoPoet over alternatives most clearly on motion interestingness and motion realism, while Lumiere was the only model to beat it on video quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on a high-quality text-video subset | VideoPoet · MSR-VTT | CLIPSIM: 0.3049 -> 0.3123 | Helped |
| Add T2I pretraining to T2V-only training (ablation) | 300M VideoPoet · MSR-VTT | CLIPSIM: 0.244 -> 0.247 | Helped |
| Add T2I pretraining to T2V-only training (ablation) | 300M VideoPoet · UCF-101 | FVD: 822 -> 1,025 | Hurt |
| Use self-supervised pretraining without paired text (ablation) | 300M VideoPoet · MSR-VTT | CLIPSIM: 0.244 -> 0.226 | Hurt |
| Remove T2I pretraining from the full task mixture (ablation) | 300M VideoPoet · SSv2 inpainting | FVD: 127 -> 95 | Helped |
| Use the full multimodal pretraining task mixture instead of T2V-only training (ablation) | 300M VideoPoet · K600 | FVD: 759 -> 729 | Helped |
| Use the full multimodal pretraining task mixture instead of T2V-only training (ablation) | 300M VideoPoet · SSv2 inpainting | FVD: 2,333 -> 127 | Helped |
| Use the full multimodal pretraining task mixture instead of T2V-only training (ablation) | 300M VideoPoet · UCF-101 | FVD: 822 -> 1,085 | Mixed |
| Scale model and training data | VideoPoet · Video and audio generation | FVD and FAD | Helped |
| Predict audio RVQ levels sequentially rather than simultaneously (ablation) | VideoPoet audio generation · Audio generation | FAD | Helped |
Surprising findings
- Pretraining on self-supervised tasks alone without text-paired data gave much worse text-to-video FVD (1,742 on MSR-VTT), showing text-paired data is necessary.
- Image-to-video conditioning and text-to-image pretraining were shared into a single task token, and omitting end-of-video tokens for images improved initial-frame quality.
Limitations
- RGB reconstruction from compressed and quantized tokens bounds visual fidelity.
- Per-frame aesthetics in static scenes lag the best baseline, attributed partly to excluding copyrighted-image sources like LAION.
- Small objects and fine-grained details with large motion remain difficult.
Experimental setup
Datasets: MSR-VTT UCF-101 Kinetics-600 Something-Something V2 DAVIS 2016
Benchmarks: MSR-VTT UCF-101 Kinetics-600 Something-Something V2 DAVIS 2016
Metrics: FVD CLIPSIM Inception Score human preference
Baselines: CogVideo MagicVideo Video LDM ModelScopeT2V InternVid VideoFactory Make-A-Video Show-1 Phenaki Lumiere Control-A-Video
Builds on: MAGVIT-v2 SoundStream T5 XL MAGVIT COMMIT
Closest papers by meaning
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.89)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.87)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.87)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.87)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.