Research map / Vision-language and multimodal / Multimodal LLMs

Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Balanced Token Pruning (BTP), a multi-stage visual token pruning method for LVLMs that shifts from diversity-based selection in early layers to attention-based selection in later layers, with calibration-selected pruning layers.

Problem

Visual token pruning in LVLMs is suboptimal because attention-based methods optimize only the current layer's output while diversity-based methods ignore local output consistency; pruning layer selection also relies on manual settings or validation performance.

Approach

Formulates a local-global objective that mixes attention-based importance (local) with a diversity term approximating later layers' token preferences (global), with weight lambda increasing across stages; uses spatial Manhattan-distance MMDP initialization for fast diversity selection, last-token attention with positional rebalancing, and calibration-based (64 samples) selection of pruning layers

Key results

• On LLaVA-1.5-7B with 128 tokens (78% reduction), BTP reaches 98% of original average performance vs 96% for DivPrune and FastV.
• On Qwen2.5-VL-7B at 25% token retention, BTP achieves 97% average relative performance, with GQA 57.2 vs 60.4 original.
• Removing spatial initialization raises latency to 0.232s (vs 0.134s with it) on LLaVA-1.5-7B, exceeding the unpruned model's latency in some settings.

Measured effects reported in the paper

Surprising findings

• DivPrune's end-to-end latency can exceed that of the uncompressed model despite lower theoretical FLOPs, because MMDP selection is O(n^2) and not GPU-friendly.
• Removing the attention rebalancing module causes a notable performance drop, attributed to positional bias of attention toward later tokens.

Limitations

• Pruning layers and lambda schedules are hand-set per model family (appendix settings differ by model), so generalization to new architectures requires re-tuning.
• Stage schedules (e.g., 50% retention per stage) are chosen manually, and the abstract's 96.7% and 22%/78% figures are not fully consistent with table entries (e.g., 98% vs 22% retention claims in the introduction).

Experimental setup

Datasets: LLaVA-655k

Benchmarks: GQA MME MMBench POPE SQA MM-Vet LLaVA-COCO

Metrics: accuracy normalized performance gain TFLOPS latency KV cache size

Baselines: FastV PyramidDrop DivPrune VTW

Builds on: FastV PyramidDrop DivPrune VTW

Closest papers by meaning

• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.95)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.94)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.94)
• SCoRe: Salience-Coverage Reduction for Vision Token Pruning in Vision-Language Models (CVPR 2026 · similarity 0.94)
• An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning (AAAI 2026 · similarity 0.93)