Research map / Vision-language and multimodal / Multimodal LLMs

QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes QuietPrune, a query-guided early token pruning method that prunes visual tokens inside the ViT using a [Q-CLS] token from a lightweight text-to-vision adapter initialized by inverting the VLM projector, with semi-structured 2x2 group pruning, trained via self-distillation.

Problem

Visual tokens make VLM inference expensive, and existing token pruning methods act late (at the LLM input or inside the LLM), ignoring ViT compute; early ViT pruning lacks textual query guidance and conflicts with 2x2 token merging.

Approach

Use an adapter obtained by inverting the pretrained projector to map the text query into a visual-domain [Q-CLS] token, whose attention to visual tokens provides relevance scores for group-wise (2x2) pruning inside the ViT, with pruned tokens aggregated into one token and only the adapter trained by self-distillation.

Key results

• At 50% LLM-side pruning, QuietPrune on InternVL3-1B retains 98.5% relative accuracy with 42.1% prefill latency reduction, versus at best 94.4% RA and 23.1% LR for late pruning baselines.
• Query-guided pruning improves relative accuracy by 4.11% (InternVL3-1B) and 5.94% (Qwen3-VL-4B) over visual-saliency-based pruning in ablation.
• Inverse-transformation initialization improves relative accuracy by 3.25% and 2.94% over random adapter initialization.

Surprising findings

• DivPrune and AIM pruning procedures are computationally expensive enough that total prefill latency exceeds the unpruned baseline in some settings.
• Unstructured pruning reaches only 81.20% and 78.36% relative accuracy versus 94.34%/95.20% for simpler structured variants in ablation, indicating positional distortion hurts badly.

Limitations

• Adapter must be trained per VLM, requiring 10K samples of training and self-distillation.
• Evaluated only on Qwen3-VL-4B, InternVL3-1B and InternVL3-8B in the main text; additional models are only in supplementary material.
• Relative accuracy gains narrow on larger models, suggesting smaller benefit for bigger VLMs.

Experimental setup

Datasets: LLaVA-Instruct ShareGPT4V ChartQA ScreenQA SynthDoG MultiVQA

Benchmarks: MMBench MMStar MMMU AI2D OCRBench HallusionBench

Metrics: relative accuracy prefill latency reduction accuracy prefill latency

Baselines: FastV DivPrune PACT AIM VisPruner SAINT-Early SAINT-Late

Builds on: EViT FastV SAINT LLaVA DynamicViT

Closest papers by meaning

• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.95)
• IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models (CVPR 2026 · similarity 0.95)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.94)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.93)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.93)
• Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.93)