Research map / Vision-language and multimodal / Multimodal LLMs

A More Word-like Image Tokenization for MLLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiVT, a density-based clustering visual tokenizer that replaces the MLP projector in LLaVA-1.5, producing adaptive, semantically disentangled visual tokens with a single similarity threshold controlling granularity.

Problem

Visual tokens from MLLM projectors are semantically entangled, highly redundant, and fixed in number per image, unlike word-like text tokens; this inflates KV-cache and latency.

Approach

Cluster ViT patch features by cosine-similarity density (greedy max-degree centroid selection with a threshold), refine assignments to nearest centroid, and aggregate each cluster via centroid-queried cross-attention with cluster-restricted masks; the token count adapts per image and the threshold can be changed at inference.

Key results

• DiVT at theta=0.75 (136.5 avg tokens) reaches 66.7 MMBench and 78.2 VQAv2 vs. 64.3 and 78.5 for the 576-token MLP baseline on LLaVA-1.5 7B.
• At ~36 tokens (theta=0.5), DiVT gets 65.0 MMB and 77.0 VQAv2 vs. TokenPacker's 62.8 and 75.0 at 36 tokens.
• At theta=0.4 (22.4 tokens), DiVT reduces KV-cache to 11.0 MB vs. 288 MB for the MLP projector and cuts prefill latency to 71.3 ms vs. 138.2 ms.

Surprising findings

• Increasing the similarity threshold beyond 0.75 (to 0.8) yields slight degradation despite more tokens, which the authors attribute to fragmentation.
• DiVT with the DINOv2 encoder underperforms the MLP baseline on MMBench (56.9 vs 59.9) and VQAv2 (74.2 vs 75.5).

Limitations

• Per-encoder thresholds need tuning (e.g., theta 0.6 for SigLIP, 0.66 for DINOv2) and DINOv2 underperforms the MLP baseline on several benchmarks.
• Tokens are formed per image by clustering, so the tokenizer adds clustering overhead, noted as heavier at higher thresholds.
• Evaluation is limited to LLaVA-1.5 style training on 7B/13B Vicuna-based models.

Experimental setup

Datasets: LLaVA-1.5 training data MMBench

Benchmarks: MMBench VQAv2 GQA MME MM-Vet TextVQA SQA-IMG POPE

Metrics: accuracy MME perception score

Baselines: MLP projector ToME FastV PruMerge+ VisionZip VisPruner ATP-LLaVA TokenPacker Resampler C-Abstractor

Builds on: LLaVA-1.5 TokenPacker ViT

Closest papers by meaning

• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
• From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.91)
• Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.91)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.90)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.90)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.90)