Research map / Vision-language and multimodal / Multimodal LLMs
Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CDPruner, a training-free, model-agnostic visual token pruning method that reformulates pruning as DPP MAP inference over a kernel conditioned on instruction relevance, to maximize conditional diversity of retained tokens.
Problem
Visual tokens in MLLMs are far more numerous than text tokens, making inference costly; existing pruning either uses attention (keeps duplicates) or feature similarity (ignores instruction relevance).
Approach
Modulate the DPP kernel of visual token cosine similarities with min-max normalized instruction relevance scores, so the selected subset is both diverse and instruction-relevant; MAP inference is solved with a fast greedy Cholesky-based algorithm.
Key results
- On LLaVA-1.5-7B with 64 tokens retained (88.9% reduction), CDPruner keeps 97.0% of original performance, vs 92.6-94.7% for the best similarity baselines.
- On LLaVA-NeXT-7B with 320 tokens, CDPruner reduces FLOPs by about 9.9x and prefill latency by 6.6x while reaching F1 87.2 on POPE (vs 86.8 unpruned).
- On Qwen2.5-VL-7B with 128 tokens retained, CDPruner keeps 85.2% of performance vs 79.9% for DivPrune.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition DPP token selection on instruction relevance (ablation) | CDPruner on LLaVA-1.5-7B · Four benchmarks; varying retained-token counts | performance | Helped |
| Use DPP-based global diversity selection instead of MMDP-style selection (ablation) | DPPruner on LLaVA-1.5-7B · Four benchmarks; varying retained-token counts | performance | Helped |
| Prune to 128 tokens with CDPruner rather than VisionZip | LLaVA-1.5-7B · LLaVA-1.5-7B, retain 128 tokens | Acc.: 62.5 vs 61.6 | Helped |
| Prune to 64 tokens with CDPruner rather than DivPrune | LLaVA-1.5-7B · LLaVA-1.5-7B, retain 64 tokens | Acc.: 61.4 vs 60.0 | Helped |
| Prune to 32 tokens with CDPruner rather than DivPrune | LLaVA-1.5-7B · LLaVA-1.5-7B, retain 32 tokens | Acc.: 60.0 vs 58.0 | Helped |
| Prune to 640 tokens with CDPruner rather than VisionZip | LLaVA-NeXT-7B · LLaVA-NeXT-7B, retain 640 tokens | Acc.: 65.1 vs 64.9 | Helped |
| Prune to 320 tokens with CDPruner rather than DivPrune | LLaVA-NeXT-7B · LLaVA-NeXT-7B, retain 320 tokens | Acc.: 64.0 vs 62.8 | Helped |
| Prune to 160 tokens with CDPruner rather than DivPrune | LLaVA-NeXT-7B · LLaVA-NeXT-7B, retain 160 tokens | Acc.: 62.8 vs 60.8 | Helped |
| Prune to 64 × 64 tokens with CDPruner rather than PDrop | LLaVA-Video-7B · LLaVA-Video-7B, retain 64 × 64 tokens | Acc.: 61.3 vs 60.3 | Helped |
| Prune to 64 × 32 tokens with CDPruner rather than DivPrune | LLaVA-Video-7B · LLaVA-Video-7B, retain 64 × 32 tokens | Acc.: 58.9 vs 57.7 | Helped |
| Prune to 64 × 16 tokens with CDPruner rather than DivPrune | LLaVA-Video-7B · LLaVA-Video-7B, retain 64 × 16 tokens | Acc.: 55.7 vs 54.9 | Helped |
| Prune to 512 tokens with CDPruner rather than FastV | Qwen2.5-VL-7B · Qwen2.5-VL-7B, retain 512 tokens | Acc.: 81.5 vs 81.1 | Helped |
Surprising findings
- Attention-based pruning degrades by over 25% on LLaVA-1.5-7B at 64 tokens, indicating text-visual attention is a poor pruning metric.
- CDPruner slightly exceeds the unpruned LLaVA-1.5 on POPE, suggesting pruning may mitigate hallucination.
Limitations
- Limited advantage on VizWiz because its questions often lack informative context for conditioning.
- Performance drop on Qwen2.5-VL is larger after pruning since visual tokens are already compressed by its projector.
Experimental setup
Datasets: POPE VQAv2 GQA VizWiz ScienceQA-IMG TextVQA MME MMBench MMBench-CN MM-Vet ChartQA AI2D
Benchmarks: VQAv2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench MMBench-CN MM-Vet ChartQA AI2D
Metrics: accuracy relative performance (Rel.) FLOPs CUDA latency KV cache GPU memory F1
Baselines: FastV PyramidDrop SparseVLM LLaVA-PruMerge+ TRIM VisionZip DART DivPrune
Builds on: DPP DivPrune FastV VisionZip DART
Closest papers by meaning
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.94)
- Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.94)
- What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.94)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.93)
- Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.