Research map / Video / Video generation
Language Model Beats Diffusion - Tokenizer is key to visual generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MAGVIT-v2, a causal video/image tokenizer with lookup-free quantization (2^18 vocabulary) that lets a masked language model beat diffusion models on ImageNet and Kinetics generation and rivals HEVC/VVC in compression.
Problem
Language models lag behind diffusion models in visual generation, and existing visual tokenizers (VQ-VAE/MAGVIT) have small vocabularies and cannot jointly tokenize images and videos well.
Approach
Lookup-free quantization (binary latents via sign function, no codebook embeddings) lets the vocabulary grow to 2^18 while improving generation; combined with a causal 3D CNN tokenizer that handles both images and videos with a shared vocabulary.
Key results
- Class-conditional ImageNet 512x512 with guidance: FID 1.91 vs 2.65 for the best diffusion baseline (VDM++), with 64 decoding steps.
- UCF-101 class-conditional FVD 58 and K600 frame prediction FVD 4.3, improving over MAGVIT (76 and 9.9).
- Rater study on MCL-JCV: raters preferred MAGVIT-v2 over HEVC and MAGVIT, and it was on par with VVC at similar bit rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MAGVIT-v2 instead of MAGVIT | MLM video generation · Kinetics-600 frame prediction | FVD: 9.9±0.3 -> 4.3±0.1 | Helped |
| use a causal tokenizer instead of the non-causal baseline | MLM video generation · Kinetics-600 frame prediction | FVD: 11.6±0.6 -> 4.3±0.1 | Helped |
| use MAGVIT-v2 instead of MAGVIT | MLM video generation · UCF-101 | FVD: 76±2 -> 58±3 | Helped |
| use MAGVIT-v2 with guidance instead of VDM++ | ImageNet 512×512 image generation · ImageNet 512×512 | FID: 2.65 -> 1.91 | Helped |
| use MAGVIT-v2 with guidance instead of VDM++ | ImageNet 256×256 image generation · ImageNet 256×256 | FID: 2.12 -> 1.78 | Helped |
| replace MAGVIT with MAGVIT-v2 | video compression · MCL-JCV | LPIPS: 0.144 -> 0.104 | Helped |
| replace MAGVIT with MAGVIT-v2 | video action recognition with tokens as inputs · SSv2 | accuracy: 57.34 -> 62.40 | Helped |
| add LFQ to MAGVIT (ablation) | image tokenization · ImageNet 128×128 | FID: 2.65 -> 2.48 | Helped |
| add a large vocabulary (ablation) | MAGVIT + LFQ image tokenizer · ImageNet 128×128 | FID: 2.48 -> 1.34 | Helped |
| add the up/downsampler modification (ablation) | image tokenizer · ImageNet 128×128 | FID: 1.34 -> 1.21 | Helped |
| increase model depth (ablation) | image tokenizer · ImageNet 128×128 | FID: 1.21 -> 1.20 | Helped |
| add adaptive normalization (ablation) | image tokenizer · ImageNet 128×128 | FID: 1.20 -> 1.15 | Helped |
Surprising findings
- Larger vocabularies improve generation with LFQ but hurt generation with standard VQ beyond small sizes.
- Video compression quality is on par with the next-generation codec VVC according to human raters, despite lower PSNR/MS-SSIM than VVC.
Limitations
- Text-to-image and text-to-video results were not available at submission time.
- Compression is not yet efficient on CPUs like standard codecs; only shown on TPUs.
- Action recognition results remain worse than state-of-the-art.
Experimental setup
Datasets: ImageNet Kinetics-600 UCF-101 MCL-JCV Kinetics-400 SSv2
Benchmarks: ImageNet 256x256 and 512x512 class-conditional generation UCF-101 Kinetics-600 frame prediction MCL-JCV SSv2 K400 K600
Metrics: FVD FID IS Elo score LPIPS PSNR MS-SSIM classification accuracy
Baselines: MAGVIT MaskGIT DiT-XL/2 ADM+Upsample Video Diffusion RIN Phenaki TATS HEVC VVC simple diffusion VDM++
Builds on: MAGVIT VQ-VAE VQGAN MaskGIT C-ViViT
Closest papers by meaning
- Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation (CVPR 2025 · similarity 0.90)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.90)
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.89)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.89)
- LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior (ICLR 2025 · similarity 0.89)
- Mimir: Improving Video Diffusion Models for Precise Text Understanding (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.