Research map / Vision-language and multimodal / Multimodal LLMs
Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoB, a training-free visual token pruning method that derives a Hausdorff-distance error bound, models prompt alignment vs. visual preservation as a bi-objective covering problem, and allocates budget between them via greedy radius trading.
Problem
Visual token pruning for MLLMs treats prompt alignment and visual preservation with static strategies, ignoring that their relative importance varies across tasks depending on prompt-visual coupling, leading to inconsistent gains.
Approach
Derives a Hausdorff-distance error bound showing a trade-off between prompt-alignment and visual-preservation covering radii governed by budget K and prompt-visual coupling eta; MoB splits the token budget into a prompt-center set (k-fold nearest-neighbor covering) and a visual-center set (farthest point sampling), so that the split implements the trade-off.
Key results
- On LLaVA-1.5-7B with 88.9% token reduction (64 tokens), MoB with eta-prior retains 96.4% of average performance, vs. 93.7% for DART.
- MoB with eta-prior retains 100.6% of vanilla LLaVA-1.5-7B performance at 66.7% reduction (192 tokens).
- On Video-LLaVA-7B at 93.4% token reduction, MoB retains 97.9% average performance, vs. 96.3% for TwigVLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the η-prior to MoB (ablation) | MoB on LLaVA-1.5-7B · 66.7% token reduction | average performance: 100.2% -> 100.6% | Helped |
| Add the η-prior to MoB (ablation) | MoB on LLaVA-1.5-7B · 77.8% token reduction | average performance: 99.2% -> 99.4% | Helped |
| Add the η-prior to MoB (ablation) | MoB on LLaVA-1.5-7B · 88.9% token reduction | average performance: 96.3% -> 96.4% | Helped |
| Use MoB with η-prior instead of TwigVLM | LLaVA-1.5-7B · 66.7% token reduction | average performance: 99.5% -> 100.6% | Helped |
| Use MoB with η-prior instead of DART | LLaVA-1.5-7B · 88.9% token reduction | average performance: 93.7% -> 96.4% | Helped |
| Use MoB with η-prior instead of DART | LLaVA-Next-7B · 88.9% token reduction | average performance: 93.9% -> 95.4% | Helped |
| Use MoB with η instead of DART | Qwen2-VL-7B · 66.7% token reduction | average performance: 97.0% -> 98.4% | Helped |
| Use MoB with η-prior instead of TwigVLM | Video-LLaVA-7B · 93.4% token reduction | average performance: 96.3% -> 97.9% | Helped |
| Increase covering fold k (ablation) | MoB · weak-coupling benchmarks | critical-region coverage | Helped |
| Increase covering fold k (ablation) | MoB · strong-coupling benchmarks | sampling diversity | Hurt |
| Prune in deeper layers instead of shallow layers (ablation) | MoB on TextVQA · TextVQA | performance-efficiency trade-off | Helped |
| Prune in deeper layers instead of shallow layers (ablation) | MoB on MME · MME | performance-efficiency trade-off | No clear effect |
Surprising findings
- Multi-objective methods such as MustDrop do not dominate single-objective methods; MustDrop is outperformed by FastV and SparseVLM on MME and POPE in latency comparisons.
- At 66.7% token reduction, MoB without the eta prior already matches or exceeds the vanilla LLaVA-1.5-7B average (100.2%).
Limitations
- Theoretical guarantees rely on Lipschitz continuity and covering regularity assumptions that may not hold for all MLLMs.
- The eta-prior variant requires an a priori estimate of prompt-visual coupling and a search for K_p and k, adding tuning overhead.
Experimental setup
Datasets: GQA MMBench MMBench-CN MME POPE VizWiz ScienceQA VQAv2 TextVQA OCRBench TGIF-QA MSVD-QA
Benchmarks: GQA MMBench MMBench-CN MME POPE VizWiz ScienceQA VQAv2 TextVQA OCRBench TGIF-QA MSVD-QA
Metrics: accuracy relative performance (% of vanilla) latency speed-up
Baselines: FastV SparseVLM MustDrop DART ToMe HiRED LLaVA-PruMerge PyramidDrop FasterVLM VisionZip TwigVLM TokenCarve
Builds on: FastV SparseVLM DART LLaVA-1.5
Closest papers by meaning
- QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.90)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.90)
- What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.90)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.90)
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.