Research map / Vision-language and multimodal / Multimodal LLMs

EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EvoComp, which trains a lightweight text-conditioned encoder-only compressor for MLLM visual tokens using labels from an evolutionary search over semantically grouped token masks, trained with GHM-C plus cosine similarity loss.

Problem

Visual tokens in MLLMs are numerous, making inference costly on high-resolution, multi-image, and edge settings; existing compression heuristics (attention, similarity) do not directly optimize output correctness, and token-level importance labels are not available in standard VL datasets.

Approach

Generate token retention labels by evolutionary search that minimizes the frozen MLLM's output loss under a one-token-per-group constraint (groups from nearest LLM vocabulary embeddings), then train a text-aware encoder-only compressor with a GHM-C loss plus a cosine-similarity separation loss between retained and pruned tokens.

Key results

• Under 88.9% visual token reduction (64 of 576 tokens) on LLaVA-1.5-7B, EvoComp (l=0) retains 94.9% of baseline average accuracy; with l=2 the 192-token setting reaches 99.3%.
• On LLaVA-NeXT-7B with 160 of 2880 tokens, EvoComp (l=0) retains 92.1% average accuracy, versus 89.6% for the next best (DART).
• On an A100, POPE prefill latency falls from 175 ms to 43 ms (4.1x) and total runtime from 246 ms to 93 ms (2.6x) on LLaVA-NeXT-7B; on a smartphone NPU, LLaVA-1.5-7B total latency speedup reaches 2.0x at 64 tokens.

Surprising findings

• The compressor trained on LLaVA-1.5-7B transfers to LLaVA-1.5-13B using the same token indices and to Qwen2.5-VL-7B without fine-tuning, remaining competitive.
• Removing the text input degrades MMB performance at 64 tokens (60.0 vs 61.9), and DPC-KNN labels outperform random labels but underperform evolutionary labels.

Limitations

• Labels require running the MLLM many times per sample (population 48 for 10 generations), making label construction costly.
• Compressor is 202M-233M parameters, which is large relative to the lightweight claim.
• Evaluation is limited to six understanding benchmarks and a single smartphone.

Experimental setup

Datasets: LLaVA-1.5 instruction tuning mixture VQAv2 GQA OKVQA OCRVQA A-OKVQA TextVQA ScienceQA MMBench POPE VizWiz

Benchmarks: GQA MMBench MMBench-CN POPE VQAv2 VizWiz

Metrics: accuracy average accuracy relative to uncompressed baseline prefill latency total latency speedup

Baselines: FastV HiRED GlobalCom^2 PruMerge VisionZip DART SparseVLM PyramidDrop MustDrop

Builds on: FastV DART GHM loss focal loss LLaVA-1.5

Closest papers by meaning

• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.92)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.91)
• UniCompress: Token Compression for Unified Vision-Language Understanding and Generation (CVPR 2026 · similarity 0.91)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information (AAAI 2025 · similarity 0.91)