Research map / Vision-language and multimodal / Multimodal LLMs

TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TransPrune, a training-free visual token pruning method for LVLMs that scores tokens by their representation transitions (magnitude and direction change across attention and FFN) combined with instruction-guided attention, reducing TFLOPs by more than half with comparable accuracy.

Problem

Visual tokens dominate LVLM inference cost, but existing pruning criteria (attention or similarity) are task-agnostic and suffer positional bias, so identifying truly important visual tokens remains difficult.

Approach

Estimate visual token importance from token transitions inside the LLM (L2-norm ratio and cosine direction change of each module's input/output), accumulated over middle layers, and fuse it with instruction-to-image attention (IGA) to progressively prune tokens without training.

Key results

• On LLaVA-1.5-7B, TransPrune-High reaches 100.0% relative accuracy at 40.8% TFLOPs (MME 1540), versus 98.8% for SparseVLM at 41.1%.
• On LLaVA-NeXT-7B, TransPrune-High retains 99.8% accuracy at 40.0% TFLOPs, outperforming PDrop (99.3% at 45.4%).
• On Qwen2.5-VL-7B, TransPrune reaches 45.1% TFLOPs with POPE 87.5 vs 86.2 for the unpruned model, while FastV is at 53.6% TFLOPs.

Surprising findings

• TransPrune-High on LLaVA-1.5-7B matches the unpruned model's accuracy (100.0%) while using about 41% of TFLOPs.
• TTV alone, which ignores the instruction, shows a notable drop on TextVQA (50.9 vs 58.2).

Limitations

• TTV alone underperforms on instruction-dependent tasks such as TextVQA, so the instruction-guided term is needed.
• Pruning layers and accumulation layers are hand-tuned (layers 7, 9, 12 with accumulation 7-12) and tuned on MME-centric analysis.
• Attention-based IGA still retains positional bias, which the method only partially mitigates.

Experimental setup

Datasets: MME MMBench SEED ScienceQA VQA-v2 POPE GQA TextVQA TGIF MSVD

Benchmarks: MME VQA-v2 SEED TextVQA ScienceQA POPE GQA MMBench TGIF MSVD

Metrics: accuracy TFLOPs latency memory

Baselines: FastV TopV PDrop ShortV SparseVLM VisionZip CDPruner

Builds on: FastV PDrop VisionZip CDPruner FlashAttention

Closest papers by meaning

• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.94)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.93)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.93)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.93)
• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.92)