Research map / Vision-language and multimodal / Multimodal LLMs

ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ZOO-Prune, a training-free visual token pruning method that scores tokens by zeroth-order finite-difference sensitivity at the projection layer and combines it multiplicatively with DivPrune-style diversity selection.

Problem

Visual tokens in VLMs are numerous and costly at inference; existing training-free pruning based on attention is unstable and diversity-only selection ignores task-relevant regions.

Approach

Estimate each visual token's sensitivity via zeroth-order random-perturbation finite differences at the lightweight multimodal projector (no backprop, no ground truth), and fuse it multiplicatively with a greedy max-min diversity score to select tokens.

Key results

• On LLaVA-1.5-7B with 64 tokens (88.9% pruned), ZOO-Prune retains 95.20% average performance vs. DivPrune 94.42% and FastV 71.10%.
• On LLaVA-NeXT-7B with 160 tokens (94.4% pruned), ZOO-Prune retains 95.4% average vs. VisionZip 90.4% and DivPrune 92.4%.
• At 160 tokens on LLaVA-NeXT-7B, ZOO-Prune reduces E2E latency by 2.30x and prefilling time by 2.59x on POPE.

Surprising findings

• Sensitivity-only selection is weaker than diversity-only on several benchmarks, notably TextVQA, while fusion gives the best average.
• Spearman correlation between token rankings at the projector and vision encoder is only moderate (0.55 MMMU, 0.49 POPE), yet the projector is used as a proxy.

Limitations

• Spearman correlation between projector and vision-encoder sensitivity rankings is only moderate, so the proxy is approximate.
• Tuning-free claims rely on hyperparameters m and h, though the paper reports robustness across ranges.
• Gains over DivPrune are often small (e.g., 0.8 points at 64 tokens on LLaVA-1.5-7B), and LLaVA-NeXT uses an extra low-rank projector factorization.

Experimental setup

Datasets: MMMU POPE

Benchmarks: GQA MMBench MME POPE ScienceQA VQAv2 TextVQA MMMU SEED-Bench

Metrics: accuracy F1 MME perception+cognition relative average performance E2E latency prefilling time

Baselines: ToMe FastV HiRED VisionZip DivPrune PyramidDrop SparseVLM

Builds on: DivPrune FastV VisionZip randomized gradient estimator

Closest papers by meaning

• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.94)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.94)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.93)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.92)
• When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA (NeurIPS 2026 · similarity 0.92)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.92)