Research map / Vision-language and multimodal / Multimodal LLMs

What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Probes visual tokens in MLLMs with EmbedLens (nearest-embedding retrieval) and finds sink, dead, and alive token groups, showing most visual computation is redundant and alive tokens align with mid-layer injection.

Problem

Visual tokens projected into LLM embedding space are poorly understood: it is unclear which carry image-specific semantics, whether they are pre-linguistic, and whether internal visual attention/FFN processing is necessary.

Approach

Introduce EmbedLens, which retrieves top-k nearest vocabulary embeddings for any hidden state, and use it with anchor-based clustering to partition visual tokens into sink, dead, and alive groups, then test pruning, sublayer skipping, and shallow-layer skipping to measure redundancy.

Key results

• Removing dead tokens (~30% of visual tokens) on LLaVA-v1.5-7B raises the average from 52.7 to 53.7, while randomly removing the same number of alive tokens drops it to 50.1.
• Skipping visual FFN (vFFN) layers lowers the CV-centric average from 54.0 to 50.6 at the 'layer decoupling' setting, while skipping the first 4-5 visual layers slightly improves the average to 53.2-53.3.
• Sink token pruning (ViT, LLM, or both) leaves the average unchanged at 52.7-52.8 on LLaVA-v1.5-7B.

Surprising findings

• Removing dead tokens improves performance, and skipping early visual layers slightly improves average accuracy.
• Color recognition in the diagnostic benchmark follows the surrounding background color rather than the object's own color.

Limitations

• Diagnostic benchmark is small (70 images) and primarily evaluated on LLaVA-style models.
• Dead-token and sink findings are reported mainly for LLaVA-v1.5; other encoders (Qwen2.5-VL, InternVL3, Qwen3-VL) do not show the same repetitive clusters.

Experimental setup

Datasets: MS COCO 2017 GQA MME MMBench MMStar TextVQA OCRBench DocVQA RefCOCO RefCOCO+ POPE HallusionBench

Benchmarks: GQA MME MMBench MMStar TextVQA OCRBench DocVQA RefCOCO+ RefCOCO POPE HallusionBench

Metrics: accuracy Acc@0.5 F1 normalized average

Baselines: LLaVA-v1.5 7B

Builds on: attention sinks logit lens CLIP

Closest papers by meaning

• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.92)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.92)
• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.92)
• From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.92)
• A More Word-like Image Tokenization for MLLMs (CVPR 2026 · similarity 0.92)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.92)