Research map / Vision-language and multimodal / Multimodal LLMs
UniTok: a Unified Tokenizer for Visual Generation and Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniTok, a unified discrete visual tokenizer trained with reconstruction plus CLIP contrastive losses, using multi-codebook quantization and attention-based projection to overcome the limited capacity of discrete tokens for both generation and understanding.
Problem
Unified visual tokenizers that serve both autoregressive generation (VQVAE) and multimodal understanding (CLIP) suffer from severe loss conflicts and degraded understanding when reconstruction and semantic supervision are combined.
Approach
Argues the conflict is not inherent; the bottleneck is the limited representational capacity of discrete token space. Multi-codebook quantization (splitting tokens into chunks with independent sub-codebooks) scales vocabulary exponentially and bottleneck dimension, and attention-based projection replaces linear factorization to preserve semantics.
Key results
- UniTok reaches 0.38 rFID and 78.6% zero-shot ImageNet accuracy (with CLIP init) at 256x256 with 16x downsampling.
- Under LlamaGen-1.4B without CFG, UniTok reduces ImageNet 256x256 gFID from 14.65 (VQGAN) to 2.51.
- UniTok-based MLLM (Llama-2-7B) reaches 76.8 VQAv2, 51.6 TextVQA, 1448 MME, and GenEval overall 0.59 versus Liquid's 0.55.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize UniTok with pretrained CLIP weights rather than random initialization (ablation) | UniTok · ImageNet | zero-shot accuracy: 70.8 -> 78.6 | Helped |
| use UniTok instead of VQGAN as the tokenizer | LlamaGen · ImageNet 256×256 generation without CFG | gFID: 14.65 -> 2.51 | Helped |
| use UniTok instead of VQGAN as the tokenizer | LlamaGen · ImageNet 256×256 generation with CFG | gFID: 2.34 -> 2.77 | Hurt |
| replace VILA-U with UniTok in the unified MLLM | Llama-2-7B unified MLLM · VQAv2 | accuracy: 75.3 -> 76.8 | Helped |
| combine reconstruction and contrastive supervision instead of using reconstruction supervision only (ablation) | UniTok tokenizer · ImageNet 256×256 validation | VQAv2: 56.33 -> 69.14 | Helped |
| combine reconstruction and contrastive supervision rather than use contrastive supervision only (ablation) | UniTok tokenizer · ImageNet 256×256 validation | VQAv2: 68.95 -> 69.14 | No clear effect |
| add contrastive supervision to reconstruction supervision (ablation) | UniTok tokenizer · ImageNet 256×256 validation | gFID: 3.59 -> 3.26 | Helped |
| use multi-codebook quantization instead of residual quantization (ablation) | 64-d tokenizer · ImageNet 256×256 validation | rFID: 3.46 -> 0.55 | Helped |
| use multi-codebook quantization instead of residual quantization (ablation) | 64-d tokenizer · ImageNet 256×256 validation | Accuracy: 58.8 -> 63.7 | Helped |
| increase the number of sub-codebooks from 1 to 8 at constant global codebook size (ablation) | MCQ tokenizer · ImageNet 256×256 validation | rFID: 1.50 -> 0.33 | Helped |
| increase the number of sub-codebooks from 1 to 8 at constant global codebook size (ablation) | MCQ tokenizer · ImageNet zero-shot classification | Accuracy: $41.0\%$ -> $46.1\%$ | Helped |
| use random initialization instead of pretrained CLIP initialization (ablation) | UniTok in LLaVA · VQAv2 | score: 69.9 -> 72.4 | Helped |
Surprising findings
- CLIP weight initialization yields better zero-shot ImageNet accuracy but worse downstream VQA than random initialization, suggesting CLIP init may be a negative prior for unified tokenizers.
- Removing reconstruction supervision from the unified tokenizer does not improve understanding, indicating the gap to CLIP stems from vector quantization rather than loss conflict.
Limitations
- Tokenizer trained only one epoch due to limited compute, which is insufficient for CLIP-style semantic convergence, especially for understanding.
- Zero-shot accuracy from random initialization is suboptimal (70.8%) relative to CLIP-initialized variant.
- GenEval results are below some diffusion and discrete diffusion baselines such as Show-o 2.0B.
Experimental setup
Datasets: DataComp-1B OpenImages ImageNet DCLM COYO LAION Mini-Gemini instruction data
Benchmarks: ImageNet VQAv2 GQA TextVQA POPE MME MM-Vet GenEval GenAI-Bench MJHQ-30K SciQA
Metrics: rFID zero-shot accuracy gFID IS precision recall VQA accuracy MME score GenEval overall FID
Baselines: VQ-GAN RQ-VAE VAR TokenFlow VILA-U CLIP SigLIP ViTamin LlamaGen Liquid Chameleon Janus
Builds on: VQVAE VQGAN CLIP Liquid LlamaGen residual quantization
Closest papers by meaning
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.92)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.91)
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.90)
- Rosetta Stone For Unified MLLMs: A Unified Tokenizer to Decipher Understanding and Generation (CVPR 2026 · similarity 0.90)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.89)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.