Research map / Vision-language and multimodal / Multimodal LLMs
VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VL-Cache, a KV cache compression method for VLMs that allocates per-layer budgets from prefill attention sparsity and scores tokens using post-vision language-token attention, retaining ~10% of KV cache with near-full accuracy.
Problem
KV cache for long visual contexts in VLMs is memory- and latency-heavy, and LLM KV cache compression methods perform suboptimally on VLMs because they ignore the modality boundary in attention and use layer budgets not adapted to the prompt.
Approach
Use attention from post-vision language tokens (a dynamic, prompt-specific window) both to compute layer-wise sparsity for prompt-adaptive budget allocation and to score which visual and language KV tokens to keep, exploiting the observed modality boundary in VLM attention.
Key results
- With 10% KV cache, VL-Cache on LLaVA-Mistral-7B reaches 100.36 CIDEr on COCO Caption versus 100.68 for full cache, while H2O gets 90.36 and PyramidKV 66.41 at 10%.
- Speed benchmark on LLaVA-family setup (up to 128K prompt, batch 1) gives up to 7.08x decoding speedup and up to 2.33x end-to-end speedup (batch 64, 2K prompt) for 100 output tokens, with 10% KV cache.
- Prefill overhead of the statistics computation is 1-6% of prefill latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VL-Cache instead of H2O at 10% KV-cache budget | LLaVA-Mistral-7B · Coco-Caption | CIDEr: 90.36 -> 100.36 | Helped |
| Use VL-Cache instead of PyramidKV at 10% KV-cache budget | LLaVA-1.6-34B · Coco-Caption | CIDEr: 116.91 -> 137.35 | Helped |
| Use VL-Cache instead of PyramidKV at 10% KV-cache budget | LLaVA-Mistral-7B · DocVQA | ANLS: 60 -> 62 | Helped |
| Use VL-Cache instead of PyramidKV at 10% KV-cache budget | LLaVA-1.6-34B · DocVQA | ANLS: 83 -> 84 | Helped |
| Use VL-Cache instead of PyramidKV at 10% KV-cache budget | LLaVA-Mistral-7B · MathVista | ACC: 38 -> 39 | Helped |
| Use VL-Cache instead of PyramidKV at 10% KV-cache budget | LLaVA-1.6-34B · MathVista | ACC: 42 -> 42 | No clear effect |
| Compress KV cache to 10% with VL-Cache | LLaVA family · Prompt length 128000, batch size 1, 100 output tokens | decoding speedup: 7.08 | Helped |
| Compress KV cache to 10% with VL-Cache | LLaVA family · Prompt length 128000, batch size 1, 100 output tokens | end-to-end speedup: 1.66 | Helped |
| Compress KV cache to 10% with VL-Cache | LLaVA family · Prompt length 8000, batch size 64, 100 output tokens | end-to-end speedup: 2.33 | Helped |
Surprising findings
- Layer attention sparsity in VLMs varies non-monotonically across layers, so pyramid-style monotonically decreasing budgets are suboptimal.
- Under the speed benchmark, maximum batch size is limited by prefill memory rather than KV cache size, so KV compression did not increase max batch size in the authors' implementation.
Limitations
- Speedup end-to-end is bounded by prefill latency, which is largely unchanged, diluting gains for short outputs.
- Evaluation limited to two LLaVA-1.6 models and three tasks; accuracy at very low budgets (1%) remains poor on some tasks.
- Concurrency gains not realized because prefill memory is the bottleneck in the current implementation.
Experimental setup
Datasets: DocVQA MathVista COCO Caption
Benchmarks: COCO Caption DocVQA MathVista
Metrics: CIDEr ANLS accuracy prefill speedup decoding speedup end-to-end speedup cache hit rate
Baselines: H2O StreamingLLM PyramidKV full KV cache
Builds on: H2O StreamingLLM PyramidKV FlashAttention
Closest papers by meaning
- Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
- TGV-KV: Text-Grounded KV Eviction for Vision-Language Models (ICML 2026 · similarity 0.92)
- PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation (NeurIPS 2025 · similarity 0.92)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.91)
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.91)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.