Research map / Vision-language and multimodal / Multimodal LLMs

Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hi-Lo Prune, a training-free visual token pruning method that hierarchically selects retained and to-be-pruned tokens and uses attention-guided prune-aware fusion to transfer information from pruned tokens to retained ones before early-layer removal.

Problem

Visual token pruning in MLLMs reduces compute but early-layer pruning discards informative visual content, degrading reasoning performance especially at high pruning ratios.

Approach

Before removing low-informative visual tokens at an early layer, a coarse-to-fine selection (pixel-level then semantic-level greedy diversity) identifies retained tokens and a set of pruning candidates; attention in the first early layers boosts information flow from candidates to retained tokens, so their content is absorbed before pruning ("look at what you'll lose").

Key results

• On Qwen2.5-VL-7B with 90% pruning, Hi-Lo Prune retains 93.51% average score versus 83.87% for CDPruner and 92.42% for DivPrune.
• On Qwen3-VL-8B at 90% pruning, MME score 2042.58 vs 1967.56 for CDPruner; FLOPs reduced 85.35% (2.576 to 0.377 TFLOPs) with 43.75% total inference time reduction.
• On LLaVA 1.5-7B at 90% pruning, average 93.88%, surpassing the second-best method by about 2.5%.

Surprising findings

• Adding PA-Fusion to CDPruner slightly reduced MME at 10% retention on Qwen3-VL-8B (1967.56 to 1959.34).

Limitations

• Gains on some benchmarks are small or negative (e.g., DocVQA and ChartQA at high pruning ratios for some models; PA-Fusion lowered MME for CDPruner at 10%).
• Evaluated mainly on Qwen-VL family and LLaVA 1.5-7B; limited model diversity.
• Relies on fixed pruning layers and ratios chosen by the authors.

Experimental setup

Benchmarks: ScienceQA POPE TextVQA DocVQA MMBench MMBench-CN MME VizWiz ChartQA

Metrics: accuracy average normalized score FLOPs total inference time prefilling time L2 distance of logits cosine similarity

Baselines: FastV DART DivPrune CDPruner PyramidDrop SparseVLM

Builds on: FastV DivPrune DART CDPruner PyramidDrop SparseVLM

Closest papers by meaning

• Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.94)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.93)
• Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.93)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.93)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.93)