Research map / Vision-language and multimodal / Multimodal LLMs

MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MMTok, a training-free visual token selection method that casts pruning as multimodal maximum coverage (text-vision plus vision-vision) solved greedily over a submodular objective, reaching near-baseline VLM accuracy at 64–192 tokens.

Problem

Vision tokens in VLMs are redundant and dominate inference cost, but existing token pruning methods use only unimodal (vision or text) signals and lack a generic selection criterion across modalities.

Approach

Formulate vision token selection as a maximum coverage problem where a subset of vision tokens must cover both the text query tokens and the full set of vision tokens; the objective is a sum of submodular coverage terms with softmax-calibrated similarities, solved by a greedy algorithm with a (1-1/e) guarantee.

Key results

• On LLaVA-1.5-7B with 64 tokens, MMTok retains 96.6% of original performance, versus 94.8% for DivPrune and 93.2% for VisionZip.
• On LLaVA-1.5-7B with 192 tokens, MMTok retains 98.7% of original performance, exceeding the fine-tuned VisionZip (98.4%).
• On LLaVA-NeXT-13B with 160 tokens (about 5.5% of 2880), MMTok retains 93.7% of average performance on the efficiency benchmark versus 89.6% for VisionZip; on POPE it achieves a 1.87x speedup while keeping 98.7% performance.

Surprising findings

• Even with zero vision tokens, LLaVA-1.5 retains about 92% of performance on MMMU and 91% on ScienceQA, so these benchmarks poorly reflect token selection quality.
• Qwen2.5-VL with zero vision tokens outputs 'No' for all POPE questions, yielding 0% F1.

Limitations

• Text-vision coverage may be insufficient when the text query is vague (e.g., captioning), requiring vision-vision coverage as a complement.
• Token budget is fixed rather than adapted to question hardness; the paper notes hard questions need more tokens.
• Results on Qwen2.5-VL are reported on 5 of 7 datasets because SQA and OCRBench are excluded for being low-IC or poorly handled by all baselines.

Experimental setup

Datasets: GQA MMBench MME POPE ScienceQA VQAv2 TextVQA MMMU SEED-Bench OCRBench MMStar

Benchmarks: GQA MMBench MME POPE ScienceQA VQAv2 TextVQA MMMU SEED-Bench OCRBench MMStar

Metrics: accuracy F1 perception+cognition score relative performance retention inference time GPU utilization memory

Baselines: FastV SparseVLM VisionZip DivPrune

Builds on: VisionZip DivPrune FastV SparseVLM

Closest papers by meaning

• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs (NeurIPS 2025 · similarity 0.91)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.91)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.90)
• SCoRe: Salience-Coverage Reduction for Vision Token Pruning in Vision-Language Models (CVPR 2026 · similarity 0.90)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.90)