Research map / Vision-language and multimodal / Multimodal LLMs

SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SemHiTok, a unified image tokenizer whose semantic-guided hierarchical codebook (SGHC) adds per-semantic-code pixel sub-codebooks on top of a frozen-after-training semantic VQ, and uses it in a 7B unified autoregressive MLLM for understanding and generation.

Problem

Unified image tokenizers for multimodal understanding and generation struggle with the trade-off between high-level semantic features and low-level pixel features, since joint training of semantic distillation and pixel reconstruction leads to suboptimal solutions, and concatenating separate tokeniz

Approach

Pretrain a semantic VQ codebook (distilling SigLIP features), then build a pixel sub-codebook for each semantic code and train only the pixel branch with reconstruction losses; the semantic and pixel indices are concatenated along the dimension and flattened into one codebook of size K×m (about 196k) so it can be used directly with next-token prediction.

Key results

• Reconstruction rFID 1.16 on ImageNet-50k at 256 resolution (0.66 at 384), versus 1.80 for VILA-U and 1.37 for TokenFlow/IBQ.
• Under LLaVA-v1.5 setting, SemHiTok reaches POPE 82.5, MME-P 1355.8, SEED 62.9, GQA 60.3, beating TokLIP and SDE among discrete tokenizers.
• As a unified 7B MLLM at 256 resolution, achieves MJHQ-30K gFID 5.40 and GenAI-Bench basic 0.83, and 384 resolution gives MMB 75.2 and MME 1993.8.

Surprising findings

• Naively concatenating separately pretrained LlamaGen and SigLIP tokens (Exp 3) hurts understanding relative to the staged SemHiTok design, suggesting raw pixel features can disturb LLM alignment.
• Norm-based VQ performed worst for semantic discretization, while EMA VQ performed best.

Limitations

• Each 256-resolution image is represented by 256 tokens, giving low generation efficiency and high computational cost.
• Chain-of-thought and advanced post-training for reasoning are not explored.
• Image editing and multi-turn conversation are not evaluated.

Experimental setup

Datasets: ImageNet COYO-700M MidJourney-style synthetic data Magpie OpenOrca Evol-Instruct EMOVA LLaVA-SFT

Benchmarks: ImageNet-50k reconstruction rFID POPE MME-P SEED GQA MMMU MMBench MME MM-Vet MJHQ-30K GenAI-Bench GenEval

Metrics: rFID accuracy gFID VQA score GenEval score DPG score codebook usage

Baselines: VILA-U SDE (MUSE-VL) TokenFlow TokLIP UniTok QLIP-B LlamaGen Janus

Builds on: VQKD VQGAN SigLIP VILA-U TokenFlow

Closest papers by meaning

• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
• Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.89)
• UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.89)
• From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.88)
• UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.88)
• Language-Guided Image Tokenization for Generation (CVPR 2025 · similarity 0.88)