Research map / Vision-language and multimodal / Human and embodied modeling
TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TokenFlow, a dual-codebook unified image tokenizer with a shared index mapping that yields discrete visual tokens usable for both multimodal understanding (surpassing LLaVA-1.5 13B) and autoregressive text-to-image generation (GenEval 0.55).
Problem
Reconstruction-targeted VQ tokenizers capture low-level detail but lack the semantic features needed for multimodal understanding, so a single discrete tokenizer serving both understanding and generation suffers a trade-off.
Approach
Decouple semantic and pixel feature learning with two encoders and two codebooks linked by a shared index mapping; indices minimize a weighted sum of semantic and pixel distances, so the same tokens carry both high-level semantics and fine pixel detail.
Key results
- TokenFlow-XL (Qwen-2.5-14B) reaches an average of 67.4 across understanding benchmarks, a 7.2% improvement over LLaVA-1.5 13B (62.9), the first discrete-input model to surpass it.
- Reconstruction rFID of 0.63 at 384x384 resolution on ImageNet 50k validation.
- Autoregressive text-to-image generation reaches GenEval 0.55 at 256x256 with 25 sampling steps (0.63 with prompt rewriting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TokenFlow-XL discrete visual input instead of LLaVA-1.5 visual input | Vicuna-13B multimodal model · Multimodal understanding average | average score: 62.9 -> 64.0 | Helped |
| Use TokenFlow-XL with Qwen-2.5-14B instead of LLaVA-1.5 | Multimodal understanding model · Multimodal understanding average | average score: 62.9 -> 67.4 | Helped |
| Use TokenFlow image tokens for autoregressive generation | Text-to-image generation model · GenEval, 256×256 | Overall: 0.53 -> 0.55 | Helped |
| Add shared mapping between semantic and pixel codebooks (ablation) | TokenFlow · ImageNet 50k validation | rFID: 8.07 -> 3.96 | Helped |
| Add multi-scale vector quantization (ablation) | TokenFlow with shared mapping · ImageNet 50k validation | rFID: 3.96 -> 2.18 | Helped |
| Initialize the semantic encoder with pretrained CLIP weights (ablation) | TokenFlow with shared mapping and MSVQ · MME-Perception | score: 1209.90 -> 1312.09 | Helped |
| Use one CLIP-distilled codebook instead of dual codebooks (ablation) | TokenFlow · ImageNet 50k validation | rFID: 2.16 -> 8.07 | Hurt |
| Use features from the final scale rather than all-scale or residual features (ablation) | TokenFlow multimodal understanding · Multimodal understanding benchmarks | overall performance | Helped |
| Use single-pass top-k/top-p sampling instead of multi-step refinement (ablation) | TokenFlow image generation · Text-to-image generation | generation coherence | Hurt |
| Increase codebook size above 32,768 (ablation) | TokenFlow tokenizer · Class-conditional image generation | Generation FID | Hurt |
| Increase codebook size (ablation) | TokenFlow tokenizer · Multimodal understanding and reconstruction | VLM score and reconstruction quality | Helped |
Surprising findings
- Reconstructing images from VQKD-quantized features (semantic-only tokenizer) gives blurred outputs, indicating semantic codebooks lose high-frequency detail.
- Using only final-scale features for understanding outperforms using all scales or residual features.
Limitations
- Generation performance slightly degrades when codebook size exceeds 32,768, so the generation experiments use a smaller codebook than the best-understanding setting.
- Text-to-image comparisons with diffusion and AR baselines rely on reported or differently configured results; FID was intentionally omitted from generation evaluation.
Experimental setup
Datasets: LAION COYO-700M ImageNet-1K LLaVA-Pretrain558K LLaVA-v1.5-mix-665K Cambrian-Alignment Cambrian-10M
Benchmarks: SEED-Bench MM-Vet POPE VQAv2 GQA TextVQA AI2D RealWorldQA MMMU MMBench MME ImageNet 50k reconstruction
Metrics: rFID PSNR SSIM accuracy GenEval overall DPG-Bench average average score across understanding benchmarks
Baselines: LLaVA-1.5 Janus VILA-U EMU3 Show-o LlamaGen VAR SDXL
Builds on: VQ-GAN VAR VQKD LLaVA-1.5 CLIP
Closest papers by meaning
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.93)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.92)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.92)
- SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation (ICLR 2026 · similarity 0.91)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.91)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.