Research map / Vision-language and multimodal / Multimodal LLMs

Rosetta Stone For Unified MLLMs: A Unified Tokenizer to Decipher Understanding and Generation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a unified visual tokenizer that hierarchically decouples shallow (pixel reconstruction) and deep (semantic alignment to CLIP/DINOv2/SAM) features in one ViT encoder, plus attention-prioritized codebook mapping, and uses it in a unified autoregressive MLLM.

Problem

Unified visual tokenizers for autoregressive MLLMs rely on pixel reconstruction and feature alignment that conflict in optimization (pixel codebook converges slowly, contrastive features poor at low-level detail), limiting both understanding and generation.

Approach

Hierarchically assign reconstruction to shallow layers and multi-teacher semantic alignment (CLIP, DINOv2, SAM) to deep layers of a single encoder; use token-similarity (key-attention) to map semantically similar patches to shared grouped pixel codebooks; add coarse-to-fine reconstruction supervision from the semantic branch.

Key results

• On ImageNet 256x256, the tokenizer reaches 0.33 rFID and 80.9% zero-shot accuracy, versus 7.6% higher than VILA-U and 2.3% above UniTok.
• The 7B unified MLLM exceeds TokenFlow-13B by 3.1% in average understanding score (63.23 vs 60.18 avg on Table 3 benchmarks).
• On MJHQ-30K the method reaches gFID 5.01 (4.93 with prompt rewriting) and GenAI-Bench advanced score 0.74 (0.86 with rewriting).

Surprising findings

• Contrastive-style models' shallow-layer features are closer to generative representations than deep layers, and the pixel codebook converges far slower than the semantic codebook.

Limitations

• Ablations on understanding use LLaVA-1.5 setting and ImageNet-only 50-epoch tokenizer training, limiting scale-related conclusions.
• Baseline comparisons mix different LLMs and resolutions (e.g., TokenFlow-13B), making the comparison not strictly controlled.

Experimental setup

Datasets: COYO SA-1B Cambrian-Alignment LAION JourneyDB Cambrian-10M ImageNet LLaVA-1.5 data

Benchmarks: ImageNet reconstruction rFID ImageNet zero-shot accuracy SEED-Bench-Img GQA MME-Perception POPE TextVQA AI2D MMMU GenAI-Bench MJHQ-30K

Metrics: rFID PSNR SSIM zero-shot accuracy VQA score gFID accuracy

Baselines: VILA-U TokenFlow UniTok SigLIP LLaVA-1.5 Show-o Janus Liquid

Builds on: VILA-U TokenFlow UniTok LlamaGen

Closest papers by meaning

• UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.90)
• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.89)
• UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.88)
• SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation (ICLR 2026 · similarity 0.87)
• Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.86)
• End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.85)