Research map / Vision-language and multimodal / Multimodal LLMs

When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines a per-token information score in VLLM decoders, finds visual token information vanishes at an "information horizon" in deep layers, and shows that adding random pruning in deep layers improves existing training-free pruning methods.

Problem

Training-free visual token pruning in VLLMs performs no better than random pruning in deep decoder layers; the cause and how to exploit it are unclear.

Approach

Quantifies a visual token's information at layer i as the drop in ground-truth first-token probability when it is removed (relative to a text-only baseline), revealing an "information horizon" beyond which all visual tokens can be removed; combining existing pruning in shallow layers with random pruning in deep layers improves efficiency-accuracy trade-off.

Key results

• DivPrune+Random retains 96.9% of Qwen2.5-VL-7B performance while pruning 50% of visual tokens (vs. 96.7% for DivPrune alone).
• On LLaVA-1.5-7B at 64 retained tokens, DART+Random reaches 91.5% relative performance and DivPrune+Random reaches 94.9%, versus 83.7% and 92.4% without random pruning.
• Efficiency: DART+Random at 64 retained tokens on LLaVA-1.5-7B TextVQA reduces latency to 0.6x of the original while improving TextVQA accuracy from 50.4% to 53.4%.

Surprising findings

• Removing low-information visual tokens at intermediate layers can improve accuracy above the unpruned model (e.g., +27.8% on MME when pruning 75% at layer 10 for LLaVA-1.5-7B).
• Pruning all visual tokens beyond a certain layer (VTW) can be nearly as harmful as the information horizon suggests only for visually complex tasks; for general tasks, deep-layer visual tokens are redundant.

Limitations

• Information metric is computed per token with a single first-token probability and requires forward passes per token, making the analysis itself costly.
• Hybrid settings and layer choices were tuned on the reported benchmarks; generality beyond the evaluated models (LLaVA-1.5, Qwen2.5-VL, InternVL3.5 in appendix) is not established.

Experimental setup

Datasets: MME TextVQA ScienceQA POPE GQA MMBench MMBench-CN InfoVQA DocVQA OCRBench OCRVQA

Benchmarks: MME ScienceQA TextVQA MMBench InfoVQA DocVQA OCRBench POPE GQA MMBench-CN OCRVQA

Metrics: accuracy relative performance retained FLOPs CUDA time latency storage

Baselines: FastV SparseVLM PDrop DART DivPrune SGL VTW

Builds on: FastV DART DivPrune VTW SparseVLM PDrop

Closest papers by meaning

• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.93)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.93)
• What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.93)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.92)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
• An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.92)