Research map / Vision-language and multimodal / Multimodal LLMs
SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
Jiahui Wang, Zuyan Liu, Yongming Rao, Jiwen Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual-head-based asymmetric KV budget allocation (SparseMM) vs SnapKV | KV cache compression on LLaVA-NeXT-Vicuna-7B · TextCaps (KV budget 256) | CIDEr: 0.65 -> 0.73 | Improved |
| add visual-head-based allocation (SparseMM) vs AdaKV | KV cache compression on LLaVA-NeXT-Vicuna-7B · OCRBench (KV budget 256) | accuracy: 0.48 -> 0.52 | Improved |
| add visual-head-based allocation (SparseMM) vs PyramidKV | KV cache compression on LLaVA-NeXT-Vicuna-7B · DocVQA (KV budget 256) | ANLS: 0.65 -> 0.68 | Improved |
| compress KV cache with SparseMM at budget 256 vs full cache | LLaVA-NeXT-Vicuna-7B · Decoding latency (16K input) | latency (ms): 52.9 -> 37.1 | Improved |
| raise input length from 8K to 32K for SparseMM decoding speedup | LLaVA-NeXT-Vicuna-7B · Decoding speedup (budget 256) | speedup: 1.16x -> 1.87x | Improved |
| SparseMM KV compression vs full KV cache at 32K input | LLaVA-NeXT-Vicuna-7B · Peak memory (32K input) | memory (GB): 32.87 -> 17.38 | Improved |
| add uniform-based cache (rho = 0.1 instead of 0) (ablation) | SparseMM on LLaVA-NeXT-Mistral-7B · OCRBench (KV budget 256) | accuracy: 0.145 -> 0.519 | Improved |
| use rho = 0.1 instead of 1.0 (score-based allocation instead of purely uniform) (ablation) | SparseMM on LLaVA-NeXT-Vicuna-7B · OCRBench (KV budget 256) | accuracy: 0.460 -> 0.522 | Improved |
| use rho = 0.1 instead of 0 (ablation) | SparseMM on LLaVA-NeXT-Vicuna-7B · OCRBench (KV budget 256) | accuracy: 0.507 -> 0.522 | Improved |
| use rho = 0.1 instead of 0 (ablation) | SparseMM on Qwen2-VL-7B-Instruct · OCRBench (KV budget 256) | accuracy: 0.809 -> 0.812 | No clear effect |
| add uniform-based cache to local window cache (ablation) | SparseMM on Qwen2-VL-7B-Instruct · MMBench (KV budget 128) | accuracy: 77.3 -> 79.3 | Improved |
| add score-preferred cache to local window + uniform cache (ablation) | SparseMM on Qwen2-VL-7B-Instruct · MMBench (KV budget 128) | accuracy: 79.3 -> 81.5 | Improved |
Later papers that compared with this method
1 later accepted paper reports SparseMM as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models (ICLR 2026 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention (ICML 2025 · similarity 0.91)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.90)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.89)
- Q Cache: Visual Attention Is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model (AAAI 2026 · similarity 0.89)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
