Research map / Vision-language and multimodal / Multimodal LLMs
What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G-Prune, a training-free graph-based visual token pruning method that scores tokens by iterative information propagation over a cosine-similarity graph, keeping both foreground and background tokens for LLaVA-NeXT.
Problem
High-resolution MLLMs such as LLaVA-NeXT use thousands of visual tokens, causing heavy computation and visual redundancy; it is unclear which visual tokens (foreground or background) must be kept for varying examples.
Approach
Treat visual tokens as nodes in a cosine-similarity graph, initialize scores with L2-norm, propagate scores over weighted links for several iterations, degree-normalize, and keep top-k tokens, so that both foreground and background representatives survive pruning without any retraining.
Key results
- On LLaVA-NeXT-8B at 50% token retention, G-Prune keeps GQA 65.34 and VQA2.0 82.54 (baseline 65.38 / 82.70) with about 1.62x speedup.
- At 90% pruning on TextVQA, G-Prune reaches 59.31 versus 38.36 for ToMe and 53.53 for FastV (baseline 65.41).
- G-Prune reports reducing 63.57% FLOPs on LLaVA-NeXT for TextVQA with a 2.34% accuracy drop; the ablation shows the graph component adds about 8 points average over L2-norm-only pruning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use G-Prune instead of ToMe at 90% token pruning | LLaVA-NeXT-8B · TextVQA | accuracy: 59.31 vs. 38.36 | Helped |
| Add L2-norm token scoring to graph propagation (ablation) | G-Prune · TextVQA | accuracy: 64.01 -> 64.05 | No clear effect |
| Add graph propagation to L2-norm token scoring (ablation) | LLaVA-NeXT-8B pruning · TextVQA | accuracy: 53.63 -> 64.05 | Helped |
| Add graph propagation to L2-norm-only scoring (ablation) | LLaVA-NeXT-8B pruning · GQA | score: 64.30 -> 64.23 | No clear effect |
| Add L2-norm initialization to graph-only token selection (ablation) | LLaVA-NeXT-8B pruning · POPE | accuracy: 87.46 -> 87.69 | No clear effect |
| Set similarity threshold to 0.9 instead of 0.5 (ablation) | G-Prune · TextVQA | accuracy: 63.87 -> 54.20 | Hurt |
| Set similarity threshold to 0.7 instead of 0.5 (ablation) | G-Prune · POPE | accuracy: 87.69 -> 87.78 | Helped |
| Increase iterations from 5 to 50 (ablation) | G-Prune · GQA | score: 64.37 -> 64.05 | Hurt |
| Increase iterations from 5 to 10 (ablation) | G-Prune · TextVQA | accuracy: 63.87 -> 63.90 | No clear effect |
Surprising findings
- Random pruning of visual tokens barely hurts MMBench but sharply degrades TextVQA, indicating task-dependent redundancy.
- L2-norm-based token selection alone performs worse than random pruning on TextVQA-style fine-grained tasks.
Limitations
- Evaluated only on a single base model (LLaVA-NeXT-8B).
- Hyperparameters (similarity threshold, iterations) chosen on the same benchmarks; performance sensitive to threshold s (drops at 0.9).
- Background tokens can be entirely pruned in some examples, which may hurt background-dependent tasks.
Experimental setup
Datasets: VQA2.0 GQA TextVQA DocVQA ChartQA POPE MME MMBench
Benchmarks: VQA2.0 GQA ChartQA TextVQA DocVQA POPE MME MMBench
Metrics: accuracy FLOPs throughput
Baselines: Random pruning ToMe FastV L2-Norm pruning
Builds on: FastV ToMe
Closest papers by meaning
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.94)
- Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.94)
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.94)
- Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.93)
- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs (NeurIPS 2025 · similarity 0.93)
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.