AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

Jiahui Wang, Zuyan Liu, Yongming Rao, Jiwen Lu

ICCV 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 16 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add visual-head-based asymmetric KV budget allocation (SparseMM) vs SnapKVKV cache compression on LLaVA-NeXT-Vicuna-7B · TextCaps (KV budget 256)CIDEr: 0.65 -> 0.73Improved
add visual-head-based allocation (SparseMM) vs AdaKVKV cache compression on LLaVA-NeXT-Vicuna-7B · OCRBench (KV budget 256)accuracy: 0.48 -> 0.52Improved
add visual-head-based allocation (SparseMM) vs PyramidKVKV cache compression on LLaVA-NeXT-Vicuna-7B · DocVQA (KV budget 256)ANLS: 0.65 -> 0.68Improved
compress KV cache with SparseMM at budget 256 vs full cacheLLaVA-NeXT-Vicuna-7B · Decoding latency (16K input)latency (ms): 52.9 -> 37.1Improved
raise input length from 8K to 32K for SparseMM decoding speedupLLaVA-NeXT-Vicuna-7B · Decoding speedup (budget 256)speedup: 1.16x -> 1.87xImproved
SparseMM KV compression vs full KV cache at 32K inputLLaVA-NeXT-Vicuna-7B · Peak memory (32K input)memory (GB): 32.87 -> 17.38Improved
add uniform-based cache (rho = 0.1 instead of 0) (ablation)SparseMM on LLaVA-NeXT-Mistral-7B · OCRBench (KV budget 256)accuracy: 0.145 -> 0.519Improved
use rho = 0.1 instead of 1.0 (score-based allocation instead of purely uniform) (ablation)SparseMM on LLaVA-NeXT-Vicuna-7B · OCRBench (KV budget 256)accuracy: 0.460 -> 0.522Improved
use rho = 0.1 instead of 0 (ablation)SparseMM on LLaVA-NeXT-Vicuna-7B · OCRBench (KV budget 256)accuracy: 0.507 -> 0.522Improved
use rho = 0.1 instead of 0 (ablation)SparseMM on Qwen2-VL-7B-Instruct · OCRBench (KV budget 256)accuracy: 0.809 -> 0.812No clear effect
add uniform-based cache to local window cache (ablation)SparseMM on Qwen2-VL-7B-Instruct · MMBench (KV budget 128)accuracy: 77.3 -> 79.3Improved
add score-preferred cache to local window + uniform cache (ablation)SparseMM on Qwen2-VL-7B-Instruct · MMBench (KV budget 128)accuracy: 79.3 -> 81.5Improved

Later papers that compared with this method

1 later accepted paper reports SparseMM as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy