Research map / Vision-language and multimodal / Multimodal LLMs
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparseVLM, a training-free, text-guided visual token pruning method for VLMs that uses text raters, rank-based per-layer sparsity, and token recycling, evaluated on LLaVA, MGM, Qwen2-VL and Video-LLaVA.
Problem
Visual tokens dominate the compute of vision-language models, and existing pruning methods either need trained networks or ignore the text query when deciding which visual tokens to remove.
Approach
Select question text tokens that are visually relevant as "raters" using their similarity to visual tokens, score visual tokens by the attention they receive from these raters inside the LLM decoder, adapt the per-layer pruning amount via the rank of the attention matrix, and merge pruned tokens into a few compact tokens via density-peak clustering instead of discarding them.
Key results
- On LLaVA-1.5-7B with 192 visual tokens (66.7% reduction), SparseVLM keeps 99.1% of vanilla accuracy, versus 88.9% for ToMe and 87.9% for FastV.
- With 64 tokens retained, SparseVLM reaches 89.3% relative accuracy versus 72.0% for FastV and 71.1% for ToMe.
- On Video-LLaVA with 194 of 2048 video tokens, SparseVLM averages 95.0% relative accuracy versus 80.3% for FastV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SparseVLM instead of PDrop at 64 tokens | LLaVA · Image understanding, Acc. (%) | accuracy: 73.4 -> 89.3 | Helped |
| use SparseVLM instead of FastV at 64 tokens | LLaVA · Image understanding, Acc. (%) | accuracy: 72.0 -> 89.3 | Helped |
| retain 192 visual tokens with SparseVLM instead of the vanilla 576-token model | LLaVA · Image understanding, Acc. (%) | accuracy: 100 -> 99.1 | Hurt |
| retain 192 visual tokens with SparseVLM instead of the vanilla 576-token model | LLaVA · Image understanding, FLOPs (T) | FLOPs (T): 4.62 -> 2.14 | Helped |
| use SparseVLM instead of FastV with 194 video tokens | Video-LLaVA · Video QA average | Acc.: 40.5 -> 47.0 | Helped |
| add token reconstruction (ablation) | LLaVA 7B · GQA, 64 tokens | accuracy: 52.2 -> 53.8 | Helped |
| add token reconstruction (ablation) | LLaVA 7B · POPE, 64 tokens | accuracy: 72.8 -> 77.5 | Helped |
| add token reconstruction (ablation) | LLaVA 7B · GQA, token counts 64/96/128/192 | Avg.: 56.2 -> 57.0 | Helped |
| add token reconstruction (ablation) | LLaVA 7B · POPE, token counts 64/96/128/192 | Avg.: 79.8 -> 82.4 | Helped |
| select relevant text raters rather than using all prompt tokens (ablation) | LLaVA · TextVQA, 64 visual tokens | accuracy: +0.8% | Helped |
| select relevant text raters rather than using all text tokens (ablation) | LLaVA · POPE, 64 visual tokens | accuracy: +2.7% | Helped |
Surprising findings
- Pruning tokens with ToMe-style merging performs worst at very low token budgets, while text-guided pruning degrades more gracefully.
Limitations
- Gains on video tasks are reported on a subset of 1000 samples per benchmark with GPT-based scoring.
- Efficiency gains depend on pruning at specific layers; the FlashAttention-compatible implementation requires an extra attention pass.
Experimental setup
Datasets: GQA MMBench MME POPE ScienceQA SEED-Bench TextVQA MM-Vet TGIF-QA MSVD-QA MSRVTT-QA ActivityNet-QA
Benchmarks: GQA MMBench MME POPE ScienceQA SEED-Bench TextVQA MM-Vet TGIF-QA MSVD-QA MSRVTT-QA ActivityNet-QA
Metrics: accuracy relative accuracy (% of upper bound) FLOPs CUDA latency GPT score
Baselines: FastV ToMe PDrop VoCo-LLaMA
Builds on: FastV ToMe PDrop FlashAttention
Closest papers by meaning
- SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.93)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.93)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.92)
- DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.92)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.