Research map / Vision-language and multimodal / Multimodal LLMs

TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TopV, a training-free visual token pruning method that formulates pruning as an optimal-transport problem solved with Sinkhorn, pruning once at prefill to stay compatible with FlashAttention and KV cache.

Problem

VLM inference is expensive because of many visual tokens; existing pruning methods (FastV, LLaVA-PruMerge, VTW) rely on attention scores or dynamic per-step pruning, which are incompatible with FlashAttention and KV cache savings.

Approach

Token importance is cast as an optimal transport problem between input visual tokens (sources) and post-LN outputs of the same layer (targets), with a visual-aware cost combining feature similarity, relative spatial Gaussian distance, and absolute central distance; Sinkhorn yields a contribution matrix whose row sums rank tokens, and pruning is done once at prefill with uniform recovery of some pr

Key results

• On InternVL2-2B at 48% visual FLOPs reduction, TopV scores 73.35 on AI2D vs 72.67 baseline and reduces dynamic memory by 61% versus FastV with about 2.1x inference efficiency relative to FastV.
• On LLaVA-v1.5-7B at 51% FLOPs reduction, TopV averages 0.39% higher accuracy than FastV (47% reduction), with 49% lower dynamic memory and 1.68x inference efficiency.
• Compared with VTW at 50% FLOPs reduction on LLaVA-v1.5-7B, TopV yields much higher NoCaps CIDEr (103.52 vs 58.37) and OCRBench (31 vs 5.1).

Measured effects reported in the paper

Surprising findings

• Attention-based pruning (FastV) can increase peak memory on LLaVA and InternVL2, because FlashAttention and KV cache incompatibility lead to memory reallocation spikes, not just token count reduction.
• Token recovery of a uniformly sampled subset of pruned tokens improves OCR performance noticeably over pruning alone.

Limitations

• Evaluation of the memory benefits is based on a single-GPU profiling subset, and the authors speculate about CUDA allocation behavior.
• Hyperparameters alpha, beta, gamma and pruning ratios are tuned per model family and task, with a mixed FLOPs-matching protocol across methods.
• Gains over baselines are modest on several tasks, and the method is not evaluated on very high-resolution or many-frame video beyond two benchmarks.

Experimental setup

Datasets: AI2D SQA_IMG MMMU MMBench POPE NoCaps OK-VQA OCRBench TGIF VideoMME

Benchmarks: AI2D SQA_IMG MMMU MMBench POPE NoCaps OCRBench OK-VQA TGIF VideoMME

Metrics: accuracy CIDEr peak memory (GB) latency token throughput FLOPs reduction ratio

Baselines: FastV VTW LLaVA-PruMerge

Builds on: FastV VTW FlashAttention Sinkhorn algorithm

Closest papers by meaning

• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.95)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.94)
• ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models (CVPR 2026 · similarity 0.94)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.94)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.93)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.93)