Research map / Vision-language and multimodal / Multimodal LLMs

An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirically analyzes attention-based vs diversity-based visual token pruning in LVLMs using effective rank and attention entropy, linking diversity to hallucination and image complexity to pruning preference, and proposes an erank-driven adaptive similarity-threshold pruning method (AgilePruner).

Problem

Visual token pruning in LVLMs reduces compute, but the feature diversity retained by attention-based and diversity-based pruning, their effect on hallucination, and when each paradigm suits different image types remain poorly characterized.

Approach

Uses effective rank (erank) and attention entropy as image- and token-set-level diagnostics to show diversity-based methods retain less diversity than intended, that higher retained diversity correlates with hallucination, and that image complexity determines which paradigm wins; then sets an adaptive similarity threshold scaled by the input's erank relative to dataset average.

Key results

• With 64 tokens on LLaVA-1.5-7B, the proposed method reaches 96.76% of full-token average versus 95.07% for VisPruner and 95.02% for DivPrune.
• On CHAIR with 64 tokens, the method reduces C_S from 57.4 (DivPrune) to 52.2 and C_I from 18.0 to 15.9 while keeping recall at 75.7.
• Adding the erank-based adaptive rule to BAT at 128 tokens raises relative performance from 96.75% to 97.91%.

Surprising findings

• Many diversity-oriented pruning methods retain substantially less feature diversity than intended, with DivPrune's erank (21.84) far from the other methods' values.
• Higher retained diversity is associated with more hallucination on CHAIR, with diversity-based methods scoring worse than attention-based ones.

Limitations

• Threshold scaling uses a single dataset-level average erank (from the LLaVA training set) and a fixed 0.01 factor, which is a simple heuristic.
• Improvements on some benchmarks are small (e.g., under 1 point) and mixed, with some inverse-adaptation and adaptive variants underperforming on specific datasets.
• Analysis is limited to a small set of LVLMs and attention/diversity paradigms; the complexity-to-method relation is shown mainly on a few datasets.

Experimental setup

Datasets: LLaVA training set MME ScienceQA POPE CHAIR COCO-C

Benchmarks: VQAv2 GQA VizWiz ScienceQA TextVQA POPE MME MMBench MMBench-CN CHAIR ChartQA AI2D

Metrics: accuracy relative performance (%) CHAIR C_S CHAIR C_I recall FLOPs latency GPU memory erank attention entropy

Baselines: FastV SparseVLM PyramidDrop VisionZip VisPruner DivPrune PruMerge+ FasterVLM ATP-LLaVA

Builds on: FastV VisionZip VisPruner DivPrune FasterVLM BAT

Closest papers by meaning

• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.93)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.93)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.93)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.92)
• When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.92)