Research map / Vision-language and multimodal / Multimodal LLMs

Hybrid Token Compression for Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HTC-VLM, which combines 4 discrete MGVQ semantic tokens with 576 continuous ViT patch tokens and compresses the 580-token hybrid sequence into one voco token, reporting 87.2% average retention over seven benchmarks versus 81.0% for VoCo-LLaMA.

Problem

Single-token visual compression for VLMs loses either high-level semantics (continuous pooling) or fine-grained detail (discrete quantization), while patch tokens incur quadratic attention cost.

Approach

Prepend a small set of discrete semantic anchor tokens (from MGVQ quantization projected to 4 tokens) to continuous patch embeddings, then compress the hybrid sequence into one trainable voco token using a text-to-vision attention mask that forces text to read only the voco latent.

Key results

• HTC-VLM retains 87.2% average performance under a one-token budget across seven benchmarks, versus 81.0% for VoCo-LLaMA.
• Continuous-only compression retains 81.0% and discrete-only retains about 33.3%, while HTC-VLM with 4 discrete tokens reaches 87.2%.
• On Qwen-VL-Chat and LLaVA-NeXT, HTC-VLM reports 94.5% and 95.7% average retention respectively, versus 86.4% and 86.6% for VoCo-LLaMA.

Surprising findings

• Discrete-only compression of 441 MGVQ tokens retains only about 33% performance, and the discrete semantic probe (v_d) underperforms the continuous probe on detail tasks.

Limitations

• Gains over VoCo-LLaMA are described as modest on the main LLaVA benchmark.
• Latency (61 ms) and memory (11.6 GB) are slightly higher than VoCo-LLaMA; the hybrid branch adds a semantic encoder overhead.
• Several baseline numbers are cited from prior papers rather than reproduced under identical conditions.

Experimental setup

Datasets: VoCo-LLaMA training data

Benchmarks: GQA VQAv2 MMBench MME POPE SEED-Bench ScienceQA-Image TextVQA MMVet

Metrics: performance retention (%) accuracy latency throughput memory

Baselines: VoCo-LLaMA Q-Former Avg. Pool ToMe FastV PDrop SparseVLM

Builds on: VoCo-LLaMA MGVQ LLaVA-1.5 SparseVLM

Closest papers by meaning

• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.93)
• DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.93)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.91)
• LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.91)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.91)