Research map / Video / Video generation
OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniTokenizer, a transformer tokenizer with decoupled window-spatial and causal-temporal attention, trained progressively (image-only then joint image-video multi-resolution) to tokenize both images and videos in one shared model, and shows its LM (VQVAE) and diffusion (VAE) variants give SOTA reconstruction and competitive generation.
Problem
Existing visual tokenizers are specialized for either images or videos, limiting flexibility and data scalability for downstream generative models; naive joint image-video training does not yield mutual benefits.
Approach
A spatial-temporal decoupled transformer (window attention spatially, causal attention temporally) with a two-stage progressive training schedule: fixed-resolution image pretraining followed by multi-resolution joint image-video training, plus KL fine-tuning to yield a continuous diffusion tokenizer.
Key results
- Ours-VQVAE reaches 1.11 rFID on ImageNet (13% better than ViT-VQGAN's 1.28) and Ours-VAE 0.69 rFID; on UCF-101 reconstruction FVD is 42 (VQVAE) and 23 (VAE) vs MAGVIT's 58.
- Autoregressive class-conditional ImageNet 256x256 generation with a 650M model reaches 7.45 FID, versus 8.81 for ViT-VQGAN.
- Ours-Latte reaches 209.2 unconditional UCF-101 FVD versus 478.0 for Latte.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use joint image-video training instead of video-only training (ablation) | Ours-VQVAE · UCF-101 and other evaluated video datasets | rFVD | Helped |
| Use progressive image pretraining followed by multi-resolution joint training instead of joint training at fixed resolution (ablation) | Ours-VQVAE · ImageNet 256 | rFID: 1.35 -> 1.11 | Helped |
| Use progressive image pretraining followed by multi-resolution joint training instead of joint training at fixed resolution (ablation) | Ours-VQVAE · Video datasets | rFVD | Helped |
| Use joint image-video training at fixed resolution instead of image-only training at fixed resolution (ablation) | Ours-VQVAE · Video datasets | rFVD | Helped |
| Use joint image-video training at fixed resolution instead of image-only training at fixed resolution (ablation) | Ours-VQVAE · ImageNet 256 | rFID: 1.28 -> 1.35 | Hurt |
| Replace joint attention and decoupled plain attention with spatial window and temporal causal attention (ablation) | Ours-VQVAE · ImageNet | rFID | Helped |
| Increase the compression rate (ablation) | Ours-VQVAE · Visual reconstruction | rFID | Hurt |
| Set latent dimension to 8 (ablation) | Ours-VQVAE · Visual reconstruction | rFID | Helped |
| Fine-tune the VQ tokenizer with KL loss as a VAE (ablation) | Ours-VQVAE · ImageNet validation | rFID: 1.11 -> 0.69 | Helped |
| Use the Ours-VAE tokenizer with DiT-XL/2 and classifier-free guidance | DiT-XL/2 · ImageNet 256×256 class-conditional generation | IS: 278.24 -> 244.23 | Hurt |
| Use the Ours-VAE tokenizer with Latte | Latte · UCF-101 unconditional video generation | FVD: 478.0 -> 209.2 | Helped |
| Scale the autoregressive model from 227M to 650M parameters | Ours with Ours-VQVAE · ImageNet 256×256 class-conditional generation | FID: 10.13 -> 7.45 | Helped |
Surprising findings
- Joint training on a fixed resolution improves video results but worsens ImageNet rFID (1.28 to 1.35), while progressive multi-resolution training recovers and improves image reconstruction to 1.11.
Limitations
- Scaling model capacity is left for future work, so the tokenizer's performance ceiling at larger sizes is untested.
- Video experiments use a limited set of datasets and a fixed 17-frame training sequence length.
Experimental setup
Datasets: ImageNet UCF-101 CelebA-HQ FFHQ Kinetics-600 Moments-in-Time Something-Something v2
Benchmarks: ImageNet CelebA-HQ FFHQ UCF-101 Moments-in-Time Kinetics-600 Something-Something v2
Metrics: rFID rFVD FID IS FVD Precision Recall
Baselines: ViT-VQGAN VQGAN DALL-E MaskGIT MAGVIT MAGVITv2 TATS CViViT Phenaki DiT-XL/2 Latte RQ-Transformer
Builds on: ViT-VQGAN VQGAN DiT Latte MaskGIT
Closest papers by meaning
- Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (ICML 2025 · similarity 0.89)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.89)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.89)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.88)
- Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction (CVPR 2025 · similarity 0.88)
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.