Research map / Vision-language and multimodal / Multimodal LLMs
SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free visual token pruning for MLLMs that selects tokens greedily by multiplying a set-coverage marginal gain with CLS-attention saliency (SCOPE score).
Problem
Visual tokens in MLLMs cause heavy computational overhead, while saliency-based pruning discards semantically important context and yields skewed attention, losing semantic completeness.
Approach
Defines a set-coverage function over cosine similarities between visual tokens, computes each unselected token's marginal coverage gain, multiplies it by its attention saliency raised to alpha, and greedily selects the token with the highest SCOPE score.
Key results
- On LLaVA-1.5 7B with 64 of 576 tokens, SCOPE retains 96.0% of vanilla average performance versus 93.5% for VisionZip and 85.1% for SparseVLM.
- With 192 tokens on LLaVA-1.5 7B, SCOPE reaches 99.5% of upper-bound average, above VisionZip (98.0%).
- On LLaVA-NeXT 7B with 160 of 2880 tokens, SCOPE keeps 95.1% average versus 92.5% for VisionZip and 86.9% for SparseVLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune to 64 tokens using SCOPE instead of VisionZip | LLaVA 1.5 7B · LLaVA 1.5 7B, retain 64 tokens | relative average performance: 93.5% -> 96.0% | Helped |
| Prune to 160 tokens using SCOPE instead of VisionZip | LLaVA-Next 7B · LLaVA-Next 7B, retain 160 tokens | relative average performance: 92.5% -> 95.1% | Helped |
| Prune to 136 video tokens using SCOPE instead of VisionZip | Video-LLaVA · TGIF, MSVD, MSRVTT, ActivityNet | relative average performance: 93.2% -> 100.5% | Helped |
| Prune to 10% of tokens using SCOPE instead of DivPrune | Qwen2-VL 7B · Qwen2-VL 7B, 10% token ratio | relative average performance: 85.90% -> 89.6% (+3.7%) | Helped |
| Add coverage to saliency-only token selection (ablation) | SCOPE on LLaVA 1.5 7B · Ablation benchmarks: GQA, MMB, MME, POPE, TextVQA | benchmark scores: 55.0 -> 58.3; 60.8 -> 61.7; 1665 -> 1698; 76.8 -> 83.9; 55.4 -> 56.6 | Helped |
| Add saliency to coverage-only token selection (ablation) | SCOPE on LLaVA 1.5 7B · Ablation benchmarks: GQA, MMB, MME, POPE, TextVQA | benchmark scores: 58.1 -> 58.3; 60.8 -> 61.7; 1687 -> 1698; 82.1 -> 83.9; 56.3 -> 56.6 | Helped |
| Use saliency-only rather than joint saliency and coverage (ablation) | LLaVA 1.5 7B · Ablation benchmarks: GQA, MMB, MME, POPE, TextVQA | benchmark scores: 58.3 -> 55.0; 61.7 -> 60.8; 1698 -> 1665; 83.9 -> 76.8; 56.6 -> 55.4 | Hurt |
| Use coverage-only rather than joint saliency and coverage (ablation) | LLaVA 1.5 7B · Ablation benchmarks: GQA, MMB, MME, POPE, TextVQA | benchmark scores: 58.3 -> 58.1; 61.7 -> 60.8; 1698 -> 1687; 83.9 -> 82.1; 56.6 -> 56.3 | Hurt |
| Prune to 160 tokens using SCOPE rather than PDrop | LLaVA-Next 7B · LLaVA-Next 7B, 160 tokens | POPE: 53.2 -> 81.3 | Helped |
| Prune to 160 tokens using SCOPE rather than the full-token baseline | LLaVA-Next 7B · LLaVA-Next 7B, 160 tokens | latency (s): 601.9 -> 188.8; 3.2× | Helped |
Surprising findings
- On some benchmarks (e.g., POPE and MM-Vet at 192 tokens on LLaVA-1.5 7B), pruned SCOPE slightly exceeds the full-token upper bound.
- Saliency-only selection performs worse than random selection on MME coverage in the coverage analysis.
Limitations
- Aggressive pruning may lose fine-grained or rare semantic information needed for detailed visual tasks.
- Evaluated only on LLaVA-1.5, LLaVA-NeXT and limited other models, so generality to other architectures is not fully validated.
Experimental setup
Datasets: GQA MMBench MME POPE ScienceQA TextVQA SEED-Bench MM-Vet TGIF MSVD MSRVTT ActivityNet
Benchmarks: GQA MMBench MME POPE ScienceQA TextVQA SEED-Bench MM-Vet TGIF MSVD MSRVTT ActivityNet
Metrics: accuracy F1 score relative performance to upper bound latency
Baselines: FastV SparseVLM VisionZip PDrop DivPrune
Builds on: FastV VisionZip DivPrune LLaVA-1.5 LLaVA-NeXT
Closest papers by meaning
- Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.93)
- What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.93)
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.92)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.92)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.92)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.