Research map / Vision-language and multimodal / Multimodal LLMs

ApET: Approximation-Error Guided Token Compression for Efficient VLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ApET, an attention-free visual token compression method that selects basis tokens, linearly reconstructs all tokens, and drops tokens with the lowest approximation error, retaining compatibility with FlashAttention.

Problem

Visual tokens in VLMs incur heavy compute; existing attention-based compression (CLS or text-vision attention) introduces positional bias and is incompatible with FlashAttention kernels.

Approach

Score visual token importance by the error of linearly reconstructing each token from a small set of sampled basis tokens (information-theoretic motivation: reconstruction MSE bounds conditional entropy), then drop low-error tokens and merge them into their most similar retained tokens, with no attention weights needed.

Key results

• On LLaVA-1.5-7B with 64 of 576 tokens (88.9% reduction), ApET keeps 95.2% of original average performance, outperforming VisionZip by about 2.5%.
• On Video-LLaVA-7B with 256 of 2048 video tokens, ApET reaches 100.4% of original average performance, versus 98.2% for VisionZip.
• On LLaVA-1.5-7B at 11.1% tokens, ApET gives 1.46x total inference speedup and 1.38x prefilling speedup on a single A100 (POPE benchmark).

Surprising findings

• ApET slightly exceeds the unpruned Video-LLaVA-7B on video benchmarks (100.4% average), which the authors attribute to token redundancy acting as noise.
• Attention-based baselines (e.g., SparseVLM, PDrop) underperform the vanilla model on Qwen2.5-VL-7B in prefilling/total time because attention weights must be recomputed.

Limitations

• The information-theoretic justification relies on a bound on reconstruction MSE, and the link to token importance is argued heuristically rather than proven.
• Image results on LLaVA-1.5 at 192 tokens and some benchmarks (e.g., VQA-Text, MMB) trail VisionZip or other baselines, so gains are not uniform.
• Token-selection hyperparameter M and sampling strategy are tuned on a few benchmarks (GQA, POPE, MME) only.

Experimental setup

Benchmarks: GQA MMBench MMBench-CN MME POPE ScienceQA VQAv2 TextVQA VizWiz TGIF MSVD MSRVTT

Metrics: accuracy relative performance vs. upper bound score total inference time prefilling time TFLOPs

Baselines: ToMe FastV SparseVLM PDrop VisionZip

Builds on: FlashAttention VisionZip FastV SparseVLM PyramidDrop

Closest papers by meaning

• Hybrid Token Compression for Vision-Language Models (CVPR 2026 · similarity 0.93)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.93)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.92)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.92)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.92)