Research map / Vision-language and multimodal / Multimodal LLMs

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LearnPruner, a two-stage visual token pruning framework for VLMs that uses a learnable pruning module after the vision encoder plus text-attention-guided pruning in the LLM's middle layer.

Problem

Visual token counts in VLMs make inference expensive; existing pruning relies on [CLS] or LLM attention scores, which are unreliable (vision encoder attention sinks, LLM attention shift toward lower-index tokens).

Approach

Replace [CLS]-attention importance in the vision encoder with a learned binary pruning module (STE-trained, frozen VLM) plus diversity-based token retention, then prune again in the LLM middle layer (layer 12) using text-to-vision attention, which is less affected by positional attention shift.

Key results

• On LLaVA-v1.5-7B with 64 tokens retained (88.9% reduction), LearnPruner reaches 96.9% relative accuracy, above the best training-free and training-based baselines in the table.
• With 32 tokens (94.4% reduction), LearnPruner keeps 94.8% relative accuracy versus 90.5% for DivPrune.
• On LLaVA-v1.5-7B at 32 tokens, prefill speedup is 2.3x and total time speedup 1.5x; on LLaVA-NeXT-7B at 160 tokens, prefill 6.0x and total 3.2x.

Surprising findings

• Using text-to-vision attention from middle layers (layer 8 onward) retains over 95% of baseline performance, while shallow and deep layer attention degrade sharply.
• Training the pruning module on only 5% of LLaVA-665K already achieves strong performance, with consistent results across data ratios.

Limitations

• Two-stage pruning requires training a learnable module per base VLM, and the approach is only evaluated on a few 7B models.
• Failure cases arise when first-stage pruning drops critical information under tight budgets in complex scenes.
• Video results use only the first 1,000 samples per benchmark and GPT-4.1 judging due to resource constraints.

Experimental setup

Datasets: LLaVA-665K GQA ScienceQA TextVQA POPE MME VQAv2 MMBench TGIF-QA MSVD-QA MSRVTT-QA

Benchmarks: GQA ScienceQA-IMG TextVQA POPE MME VQAv2 MMBench MMBench-CN TGIF-QA MSVD-QA MSRVTT-QA

Metrics: accuracy relative accuracy (RelAcc.) prefill time total time TFLOPs KV cache size GPU memory

Baselines: FastV SparseVLM DivPrune DART VisPruner VisionZip TwigVLM ATP-LLaVA

Builds on: FastV VisionZip TwigVLM Dynamic-LLaVA

Closest papers by meaning

• IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models (CVPR 2026 · similarity 0.94)
• ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models (CVPR 2026 · similarity 0.94)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.94)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.94)
• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.94)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.94)