Research map / Vision-language and multimodal / Multimodal LLMs
Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes training-free, prompt-aware vision token reduction for MLLMs: G-Search finds per-layer keeping rates that cut compute without accuracy loss, and P-Sigmoid fits a parametric sigmoid schedule under a fixed budget to maximize accuracy.
Problem
MLLMs produce many vision tokens that scale quadratically with image resolution, making inference expensive; prior prompt-aware reduction rules are handcrafted per model and do not optimize for limited compute budgets.
Approach
Vision-token attention rankings are stable across layers after the first, so the set of essential top tokens cannot grow with depth; this enables a greedy layer-by-layer search (G-Search) for keeping rates, and a sigmoid fit (P-Sigmoid) whose single shape parameter is tuned by Bayesian optimization to maximize accuracy under a budget.
Key results
- G-Search reduces TFLOPs by 16.9%, 30.4%, 35.9%, and 49.2% on InternVL2-1B, 2B, 4B, 8B with minor accuracy change (about 0.2% average drop overall, up to 2.3x speedup).
- On LLaVA-1.5-7B at 87.5% vision token reduction, P-Sigmoid outperforms FastV by +3.38 average accuracy points, rising to +5.46 at about 94% reduction.
- On InternVL2-8B at 87.5% reduction, P-Sigmoid achieves +7.69 average accuracy over FastV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply G-Search (searched keeping rates) to reduce vision tokens | LLaVA-1.5-7B · 12 benchmarks | Avg. acc. and TFLOPs: 48.77 vs 48.97 avg acc at 3.95 vs 9.18 TFLOPs | Helped |
| Apply G-Search reduction | InternVL2-8B · 12 benchmarks | Avg. acc. and TFLOPs: 70.10 vs 70.83 avg acc; TFLOPs 12.24 vs 24.10 | Helped |
| Apply G-Search reduction | InternVL2-1B/2B/4B · 12 benchmarks | Avg. acc.: 59.19 vs 59.85 (1B); 61.22 vs 61.94 (2B); 67.65 vs 68.16 (4B) | No clear effect |
| Use prompt-aware VTW reduction at 50% | InternVL2 family and LLaVA-1.5-7B · 12 benchmarks | Avg. acc.: e.g. 52.51 vs 70.83 on InternVL2-8B; 37.84 vs 61.94 on InternVL2-2B | Hurt |
| Apply G-Search on top of prompt-agnostic TokenPacker | TokenPacker (LLaVA-1.5-7B) · 12 benchmarks | Avg. acc. and TFLOPs: TFLOPs 3.27 to 2.12 with avg acc 47.60 to 47.68 | Helped |
| Apply G-Search on top of prompt-agnostic DeCo | DeCo (LLaVA-1.5-7B) · 12 benchmarks | Avg. acc. and TFLOPs: 46.71 vs 46.97 avg acc (-0.26); TFLOPs 3.26 to 2.16 | Mixed |
| Replace FastV at 87.5% reduction with P-Sigmoid budget-matched keeping rates | LLaVA-1.5-7B · 12 benchmarks at similar TFLOPs | Avg. acc.: +3.38 (46.52 vs 43.14) | Helped |
| Replace FastV at 87.5% reduction with P-Sigmoid budget-matched keeping rates | InternVL2-8B · 12 benchmarks at similar TFLOPs | Avg. acc.: +7.69 (62.86 vs 55.17) | Helped |
| Replace FastV at 87.5% reduction with P-Sigmoid budget-matched keeping rates | InternVL2-1B, 2B, 4B · 12 benchmarks at similar TFLOPs | Avg. acc.: +4.67, +3.88, +6.36 | Helped |
| Apply P-Sigmoid keeping-rate schedule on video (InternVL2-8B, 87.5% budget) | InternVL2-8B video · MVBench and Video-MME | Avg. acc. on video benchmarks: MVBench 64.70 vs 63.88 and Video-MME 52.48 vs 51.63 vs FastV | Helped |
| Use another MLLM's G-Search keeping rates (LLaVA-1.5-7B strategy) instead of self-searched (ablation) | InternVL2-8B · 12 benchmarks | Avg. acc.: 69.49 vs 70.10 | Hurt |
| Train or finetune LLaVA-1.5-7B with G-Search reductions (ablation) | LLaVA-1.5-7B · 12 benchmarks | Avg. acc.: 48.365 trained vs 48.965 pretrained | No clear effect |
Surprising findings
- Reduction strategies searched for one MLLM transfer poorly to others; e.g., LLaVA-1.5-7B's keeping rates lower InternVL2-8B accuracy from 70.10 to 69.49.
- Training or fine-tuning LLaVA-1.5-7B with the searched reductions gave no performance boost for Scenario I.
Limitations
- Requires attention scores in the prefilling stage, so it is incompatible with FlashAttention-style I/O-aware kernels for prefill.
- Searched strategies are tuned on image data and are not optimal for video, where more redundancy exists.
- Token removal decisions are made only during prefilling, not adapted during generation.
Experimental setup
Datasets: LLaVA training data MME MMBench GQA MMMU MathVista AI2D TextVQA ChartQA DocVQA POPE RealWorldQA
Benchmarks: MME MMBench GQA MMMU MathVista AI2D TextVQA ChartQA DocVQA POPE RealWorldQA HallusionBench
Metrics: accuracy relaxed accuracy ANLS F1 TFLOPs prefilling time memory cost
Baselines: FastV PDrop VTW TokenPacker DeCo FlashAttention2
Builds on: FastV StreamingLLM Bayesian Optimization
Closest papers by meaning
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.92)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.92)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.91)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.91)
- VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.