Research map / Vision-language and multimodal / Multimodal LLMs

Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VisionDrop, a training-free, visual-only (image-to-image attention) progressive token pruning and merging framework spanning the visual encoder and LLM layers for LVLMs.

Problem

Visual tokens in LVLMs are numerous and costly; existing in-LLM pruning methods rely on text-guided attention, which the authors argue is unreliable due to causal, semantic, and spatial cross-modal misalignment.

Approach

Score visual tokens using only visual-to-visual attention (no text queries), apply this progressively at multiple stages across the vision encoder and LLM, and merge non-dominant tokens into contextual tokens via key similarity.

Key results

• On LLaVA-1.5-7B with 32 retained tokens (94.4% reduction), VisionDrop keeps 91.46% of original average performance, vs 90.50% for the best baseline (VisPruner).
• On LLaVA-NeXT-7B with 320 retained tokens, VisionDrop reaches 95.71% average performance and reports 2.7x latency reduction and 6x FLOPs reduction (vs. 46.25T to 7.70T FLOPs).
• On LLaVA-NeXT-7B with 160 tokens, VisionDrop achieves 92.06%, 1.71 points above the second-best approach.

Surprising findings

• Removing the causal mask in visual-to-visual attention yields comparable or slightly better performance than the default causal mask, suggesting text-guided LLM pruning is questionable.
• Visual-only attention scoring outperforms text-guided (image-to-text) scoring in a controlled PyramidDrop comparison, with the gap widening under aggressive pruning.

Limitations

• Video results use only the first 1,000 samples per benchmark due to API usage limits, and the judge is GPT-3.5-Turbo.
• Hyperparameters such as stage boundaries and retention ratios are configured per budget, with the second-stage retention set to 1.5x (image) or 3x (video) the final count.
• The paper claims applicability to medical and remote sensing domains but does not evaluate them.

Experimental setup

Datasets: GQA MMBench MMBench-CN MME POPE ScienceQA VQAv2 TextVQA VizWiz TGIF MSVD MSRVTT

Benchmarks: GQA MMBench MMBench-CN MME POPE ScienceQA VQAv2 TextVQA VizWiz TGIF MSVD MSRVTT

Metrics: accuracy average relative performance (%) quality score (0-5) FLOPs latency memory

Baselines: FastV PyramidDrop SparseVLM-v1.5 VisionZip VisPruner

Builds on: PyramidDrop FastV VisionZip SparseVLM

Closest papers by meaning

• One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.93)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.93)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.93)
• Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.92)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.92)