Research map / Vision-language and multimodal / Multimodal LLMs
DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DivPrune, a training-free, calibration-free visual token pruning method that formulates token selection as a Max-Min Diversity Problem over cosine distances, retaining diverse visual tokens in LMMs.
Problem
Visual tokens in large multimodal models add thousands of tokens, raising LLM inference latency and memory; existing pruning methods either need calibration/fine-tuning or use attention-based importance scores that leave redundant tokens at high pruning ratios.
Approach
Cast visual token pruning as a Max-Min Diversity Problem: greedily select a subset of tokens that maximizes the minimum cosine distance among selected tokens, then discard the rest before the LLM, with no calibration or fine-tuning.
Key results
- On LLaVA 1.5-7B at ~15.6% TFLOP, DivPrune reaches 56.85 GQA EM and 59.19 MMBench accuracy versus 38.94 and 21.31 for VTW, and 38.73 and 20.62 for FastV.
- On LLaVA-NeXT-Video-7B at 14.1% TFLOP, DivPrune scores 45.90 on ActivityNet accuracy vs 33.91 for FastV and 26.84 for VTW, and cuts E2E latency from 4.37s to 3.39s.
- Min-Max selection (the opposite objective) performs worst with 46.53 average vs 62.34 for DivPrune in the ablation on LLaVA 1.5-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DivPrune instead of attention-based FastV | LLaVA 1.5-7B · GQA | EM: 38.73 -> 56.85 | Helped |
| use DivPrune instead of FastV | LLaVA 1.5-7B · MMBench | accuracy: 20.62 -> 59.19 | Helped |
| use DivPrune instead of VTW | LLaVA 1.5-7B · POPE | F1: 25.35 -> 86.02 | Helped |
| use DivPrune instead of the original unpruned model | LLaVA 1.5-7B · POPE | F1: 85.84 -> 86.02 | Helped |
| prune video tokens with DivPrune instead of FastV | LLaVA-NeXT-Video-7B · ActivityNet | accuracy: 33.91 -> 45.90 | Helped |
| prune video tokens with DivPrune instead of the original unpruned model | LLaVA-NeXT-Video-7B · VideoChatGPT dataset | E2E latency: 4.37 -> 3.39 | Helped |
| prune after decoder layer 1 rather than before the first decoder layer (ablation) | DivPrune on LLaVA 1.5-7B · Average across MMB, MMMU, POPE, SQA | average score: 62.34 -> 60.44 | Hurt |
| prune after decoder layer 2 rather than before the first decoder layer (ablation) | DivPrune on LLaVA 1.5-7B · Average across MMB, MMMU, POPE, SQA | average score: 62.34 -> 56.99 | Hurt |
| prune after decoder layer 3 rather than before the first decoder layer (ablation) | DivPrune on LLaVA 1.5-7B · Average across MMB, MMMU, POPE, SQA | average score: 62.34 -> 38.60 | Hurt |
| replace cosine distance with L1 distance (ablation) | DivPrune on LLaVA 1.5-7B · Average across MMB, MMMU, POPE, SQA | average score: 62.34 -> 61.94 | No clear effect |
| replace cosine distance with L2 distance (ablation) | DivPrune on LLaVA 1.5-7B · Average across MMB, MMMU, POPE, SQA | average score: 62.34 -> 62.22 | No clear effect |
| replace max-min diversity selection with random selection (ablation) | DivPrune on LLaVA 1.5-7B · Average across MMB, MMMU, POPE, SQA | average score: 62.34 -> 56.66 | Hurt |
Surprising findings
- Pruning visual tokens with DivPrune slightly improves the original model's MMMU accuracy on LLaVA 1.6-7B and POPE on LLaVA 1.5-7B.
- Attention-based methods FastV and VTW collapse on captioning (COCO CIDEr drops ~95%) at ~15% TFLOP.
Limitations
- Prefill time is about 6-7% longer than FastV/VTW due to distance computation.
- Gap to FitPrune (calibration-based) is slight at high TFLOP ratios, where FitPrune marginally outperforms it.
- Some baselines (VTW, FitPrune, M^3) could not be configured to exactly match TFLOP ratios, which may advantage baselines.
Experimental setup
Datasets: COCO-2017 Flickr30k GQA MMBench MME MMMU NoCaps OKVQA POPE ScienceQA-IMG SEED-Bench ActivityNet
Benchmarks: COCO-2017 Flickr30k GQA MMBench MME MMMU NoCaps OKVQA POPE ScienceQA-IMG SEED-Bench ActivityNet
Metrics: CIDEr EM Accuracy F1 P-score GPT-assisted score WUPS TFLOP ratio max GPU memory prefill time
Baselines: FastV PruMerge VTW FitPrune M^3
Builds on: FastV PruMerge FitPrune VTW LLaVA-1.5
Closest papers by meaning
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.91)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.91)
- Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.91)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.91)
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.90)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.