Research map / Vision-language and multimodal / Multimodal LLMs

Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HoloV, a plug-and-play visual token pruning method for MLLMs that allocates the pruning budget across spatial crops using intra-crop diversity plus [CLS] attention, retaining holistic context instead of only highlighted tokens.

Problem

Attention-first visual token pruning (text-vision or [CLS] attention) in MLLMs keeps semantically similar, highlighted tokens, causing sharp performance drops at high pruning ratios.

Approach

Scores visual tokens by combining intra-crop similarity variance (diversity) with [CLS] attention, then adaptively allocates the retained-token quota across spatial crops before the LLM decoder, so pruning is done pre-LLM and remains compatible with Flash-Attention.

Key results

• LLaVA-1.5-7B with HoloV retains 95.8% of original average performance after pruning 88.9% of visual tokens (64 tokens), versus 93.9% for the next best method DART.
• On LLaVA-NeXT-7B with 320 of 2880 tokens kept (88.9% pruned), HoloV reaches 95.6% average retention, versus 93.9% for DART.
• On POPE at 90% pruning on LLaVA-1.5-7B, HoloV reaches 95.7% accuracy retention versus 92.7% for HiRED and 66.8% for FastV, with 42.7% inference time reduction.

Measured effects reported in the paper

Surprising findings

• A random crop-based masking strategy outperforms FastV on more than half of the benchmarks, suggesting holistic context matters more than attention salience.
• Using only the global thumbnail yields strong general perception results, while local crops alone excel at fine-grained tasks such as TextVQA and POPE.

Limitations

• Gains over the strongest baselines are small on several benchmarks (e.g., 95.8% vs 93.9%), and crop-count ablation shows performance sensitive to pruning rate at 88.9%.
• Theoretical analysis relies on assumptions (C1-C2) and some proofs are loosely stated, with the claimed bounds not empirically validated.
• Visual context refetching component is described only briefly in the main text and its contribution is not isolated in the reported ablations.

Experimental setup

Datasets: GQA MMBench MMBench-CN MME POPE ScienceQA VQAv2 TextVQA VizWiz MM-Vet MSVD-QA MSRVTT-QA

Benchmarks: GQA MMBench MMBench-CN MME POPE ScienceQA VQAv2 TextVQA VizWiz MM-Vet MSVD-QA MSRVTT-QA

Metrics: accuracy average performance retention (%) latency prefill time GPU memory

Baselines: ToMe FastV SparseVLM HiRED LLaVA-PruMerge PDrop MustDrop FasterVLM GlobalCom$^2$ VisionZip DART

Builds on: FastV FasterVLM DART VisionZip

Closest papers by meaning

• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.94)
• What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.94)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.93)
• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.93)
• Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.93)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs (NeurIPS 2025 · similarity 0.93)