Research map / Vision-language and multimodal / Multimodal LLMs
Towards Semantic Equivalence of Tokenization in Multimodal LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeTok, a dynamic density-peak clustering vision tokenizer that groups patch features into a variable number of semantic-complete tokens, and SeTokim, an LLaMA-2-7B-based MLLM using these tokens for understanding, generation, editing and referring segmentation.
Problem
Existing vision tokenizers (fixed patch grids, learnable queries) fragment objects across tokens, breaking semantic integrity and weakening fine-grained vision-language alignment in MLLMs.
Approach
Replace fixed patch or query tokens with a dynamic, density-peak-based hard clustering of ViT patch features into a variable number of semantic units, merged by transformer token mergers, trained with image reconstruction plus a concept-level image-text contrastive loss.
Key results
- SeTok reaches rFID 2.07 and 75.4 top-1 ImageNet classification, versus 72.6 for TiTok.
- SETOKIM achieves 86.9 CIDEr on Flickr30K and 65.6 GQA, and 68.0 / 72.4 / 61.2 on refCOCO+ val/testA/testB.
- Dynamic SeTok tokenization (about 8.2 TFLOPs) outperforms fixed 256/32/64-token connectors (e.g., 80.6-83.4 Flickr30K) in the connector comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SeTok semantic-equivalent tokens instead of patch-level tokens | Setokim · GQA | accuracy: performance improvement on GQA by 3.6% | Helped |
| Use hard rather than soft clustering (ablation) | SeTok · OK-VQA | accuracy: 58.9 -> 60.2 | Helped |
| Use dynamic clustering rather than fixed cluster counts (ablation) | SeTok · Flickr30K | CIDEr: 82.1 / 83.4 / 84.1 / 85.1 -> 86.9 | Helped |
| Add concept-level image-text contrastive loss (ablation) | SeTok · VQA^v2 | accuracy: 65.8 -> 78.5 | Helped |
| Add 2D positional encoding (ablation) | SeTok · ImageNet | rFID: 3.56 -> 2.07 | Helped |
| Add inter-cluster Transformer (ablation) | SeTok · ImageNet | rFID: 7.91 -> 2.07 | Helped |
| Add inner-cluster Transformer (ablation) | SeTok · ImageNet | rFID: 6.25 -> 2.07 | Helped |
| Use Token Merger rather than simple average visual representations (ablation) | SeTok · ImageNet | rFID: 8.64 -> 2.07 | Helped |
| Keep the vision encoder trainable rather than frozen (ablation) | SeTok · ImageNet | rFID: 123.6 -> 2.07 | Helped |
| Freeze the vision encoder (ablation) | SeTok · Flickr30K | CIDEr: 85.4 -> 86.9 | No clear effect |
| Use only reconstruction loss instead of both losses (ablation) | SeTok · Flickr30K | CIDEr: 78.1 | Hurt |
| Use only concept-level image-text contrastive loss instead of both losses (ablation) | SeTok · Flickr30K | CIDEr: 83.6 | Hurt |
Surprising findings
- Hard clustering outperformed soft clustering in the reported ablation (86.9 vs 86.7 Flickr30K, 60.2 vs 58.9 OK-VQA).
- Freezing the vision encoder during SeTok training causes reconstruction to fail (rFID 123.6 vs 2.07) while leaving vision-language scores nearly unchanged.
Limitations
- Evaluated only at 7B scale; larger backbones left for future work.
- Input resolution limited to 384x384, weak on high-resolution fine-grained content.
- Hallucinations remain on POPE despite improvements.
Experimental setup
Datasets: ImageNet-1K OpenImages CC12M LAION-aesthetics-12M ALLaVA-Caption-4V InstructPix2Pix LLaVA-595K MSCOCO Visual Genome SlimPajama ALLaVA-Instruct-4V ShareGPT4V
Benchmarks: Flickr30K VQAv2 OK-VQA GQA POPE MME MM-Vet ImageNet MSCOCO MagicBrush MA5K EVR
Metrics: CIDEr accuracy FID rFID top-1 accuracy CLIP-image similarity L1 LPIPS gIoU cIoU
Baselines: InstructBLIP Qwen-VL-Chat Emu DreamLLM LLaVA-1.5 NExT-GPT SEED-X LaVIT Unified-IO-2 CM3Leon Chameleon MagicBrush
Builds on: LLaVA TiTok SAM LISA Chameleon
Closest papers by meaning
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.90)
- A More Word-like Image Tokenization for MLLMs (CVPR 2026 · similarity 0.90)
- Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.89)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.89)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.88)
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.