Research map / Vision-language and multimodal / Multimodal LLMs

Variation-aware Vision Token Dropping for Faster Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes V²Drop, a training-free inner-LLM visual token dropping method that progressively removes vision tokens with the smallest layer-to-layer representation variation (L2 norm), avoiding attention weights and positional bias.

Problem

Inner-LLM visual token compression methods rely on attention weights, which induce positional bias toward later tokens and are incompatible with efficient attention operators like FlashAttention, hurting LVLM inference efficiency on high-resolution images and long videos.

Approach

Measure each vision token's variation across adjacent LLM layers (L1/L2/cosine) as an intrinsic importance signal and progressively drop low-variation "lazy tokens" at three pruning layers, without computing attention scores.

Key results

• On LLaVA-1.5-7B with 192 retained tokens, V²Drop keeps 97.6% of average performance, ahead of PDrop at 96.0%.
• On LLaVA-OV-7B video benchmarks at 25% retention, V²Drop reaches 98.6% of original performance, versus 97.7% for DyCoke at 30%.
• V²Drop reduces LLM generation latency by 31.5% on image understanding and 74.2% on video understanding, per the abstract and efficiency table.

Surprising findings

• SparseVLM increases GPU peak memory by 54.8% on MVBench because of its merging strategy and explicit attention computation.
• Attention-guided methods such as FastV and PDrop retain later-positioned tokens regardless of content, a positional bias that V²Drop avoids.

Limitations

• The theoretical first-order justification relies on assumptions (bounded, non-degenerate Jacobians, small layer changes) that are argued informally rather than verified empirically.
• Pruning schedule and layer choices (e.g., layers 3, 17, 22 for LLaVA-1.5) are hand-selected per retention ratio.

Experimental setup

Datasets: GQA MMBench MME POPE ScienceQA TextVQA AI2D MMStar MVBench VideoMME

Benchmarks: GQA ScienceQA TextVQA POPE MME MMBench AI2D MMStar MVBench VideoMME

Metrics: accuracy average relative performance LLM generation latency throughput GPU peak memory

Baselines: ToMe FastV HiRED LLaVA-PruMerge SparseVLM PDrop DART DyCoke Random

Builds on: FastV SparseVLM PDrop

Closest papers by meaning

• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.93)
• DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.93)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.93)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.93)
• VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration (ICLR 2026 · similarity 0.93)
• Hybrid Token Compression for Vision-Language Models (CVPR 2026 · similarity 0.92)