Research map / Vision-language and multimodal / Multimodal LLMs
Each Complexity Deserves a Pruning Policy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoPrune, a training-free token pruning method that uses cross-modal mutual information from attention to set per-sample budget-constrained logistic retention curves for visual tokens in LVLMs.
Problem
Fixed or layer-heuristic visual token pruning schedules in large vision-language models do not adapt to sample/task complexity and do not reliably meet a global compute budget.
Approach
Measure mutual information between visual and textual tokens from attention maps as a complexity indicator, map it to the slope and inflection point of a logistic retention curve, and renormalize the curve so its area equals a user-specified token or FLOPs budget.
Key results
- On LLaVA-1.5-7B at 64 retained tokens (89% pruned), AutoPrune keeps 96.7% of original performance vs. 87.6% for PDrop, with 23.2% FLOPs.
- On LLaVA-NeXT-7B at 160 tokens, AutoPrune retains 94.9% vs. 86.7% for FasterVLM.
- On Senna autonomous driving planning at 25% retention, AutoPrune reaches 111.23% relative accuracy, above PyramidDrop's 98.89%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoPrune instead of PDrop at 64 tokens | LLaVA-1.5-7B · 64 tokens | Ratio: 87.6% -> 96.7% | Helped |
| Use AutoPrune instead of PDrop at 128 tokens | LLaVA-1.5-7B · 128 tokens | Ratio: 95.6% -> 98.1% | Helped |
| Use AutoPrune instead of PDrop at 192 tokens | LLaVA-1.5-7B · 192 tokens | Ratio: 96.8% -> 99.0% | Helped |
| Use AutoPrune instead of FasterVLM at 160 tokens | LLaVA-NeXT-7B · 160 tokens | Ratio: 86.7% -> 94.9% | Helped |
| Use AutoPrune instead of PyramidDrop at 32/128 retention | Senna · nuScenes, 32/128 (25%) retention | Performance retention: 98.89% -> 111.23% | Helped |
| Replace mutual information with cosine similarity (ablation) | AutoPrune · TextVQA, 64 tokens | accuracy: 57.1 -> 55.8 | Hurt |
| Replace mutual information with average attention (ablation) | AutoPrune · GQA, 128 tokens | accuracy: 59.9 -> 58.4 | Hurt |
| Use a static logistic schedule instead of complexity-adaptive mutual information (ablation) | AutoPrune · TextVQA, 128 tokens | accuracy: 57.4 -> 55.9 | Hurt |
| Replace the logistic retention curve with a linear curve (ablation) | AutoPrune · TextVQA, 64 tokens | accuracy: 57.1 -> 54.1 | Hurt |
| Replace the logistic retention curve with tanh (ablation) | AutoPrune · TextVQA, 64 tokens | accuracy: 57.1 -> 56.6 | Hurt |
| Replace the logistic retention curve with an exponential curve (ablation) | AutoPrune · GQA, 128 tokens | accuracy: 59.9 -> 56.3 | Hurt |
Surprising findings
- On the Senna driving VLA, the pruned model sometimes outperforms the full unpruned model, which the authors attribute to noisy visual tokens.
- The authors observe that deeper layers can occasionally retain more critical tokens than shallower ones, contrary to the general decreasing-importance assumption.
Limitations
- Deeper layers can sometimes retain more critical tokens, so monotonic decay assumption is imperfect.
- Mutual information is computed from attention under a uniform text prior, an approximation of true joint distribution.
- Some LLaVA-NeXT baseline results were taken from prior publications while others were reproduced in supplementary material.
Experimental setup
Datasets: MME MMB ScienceQA GQA TextVQA VQAv2 POPE nuScenes
Benchmarks: MME MMB ScienceQA GQA TextVQA VQAv2 POPE nuScenes planning
Metrics: accuracy performance retention ratio FLOPs reduction
Baselines: ToMe FastV SparseVLM PDrop FasterVLM VisionZip PyramidDrop
Builds on: PyramidDrop FastV SparseVLM
Closest papers by meaning
- Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving (CVPR 2026 · similarity 0.93)
- LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.92)
- When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA (NeurIPS 2026 · similarity 0.92)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.92)
- SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.92)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.