Research map / Architectures and efficiency / Attention and transformers
PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PrefixKV, which allocates per-layer KV cache sizes for LVLMs by binary-searching a global cumulative-importance threshold so each layer keeps maximal top-ranked KV information under a fixed budget.
Problem
KV cache compression for large vision-language models usually keeps the same cache size in every layer, ignoring layer-wise differences in KV importance distributions, which causes large information loss in some layers and degraded generation.
Approach
Defines the cache as a top-importance "prefix" per layer and reframes per-layer size allocation as searching for a global cumulative-priority threshold p (via binary search) that meets the overall budget; the resulting configuration is estimated offline on about 10 random samples and reused at inference.
Key results
- On LLaVA-Description at 20% budget, LLaVA-1.5-7B PPL 3.69 vs 12.4 (Pyramid) / 48.3 (H2O) / 14.0 (Elastic), close to uncompressed 3.20.
- Under 20% budget with batch size 16 on LLaVA-1.5-7B, PrefixKV gives 1.8x throughput speedup over full cache on an A100.
- Offline estimation from random samples achieves performance comparable to online per-sample binary search; binary search converges in 7-12 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive layer-wise prefix KV retention instead of Pyramid | LLaVA-1.5-7B · LLaVA-Description, 20% cache budget | PPL: 12.4 -> 3.69 | Helped |
| use adaptive layer-wise prefix KV retention instead of H₂O | LLaVA-1.5-7B · MM-Vet, 20% cache budget | ROUGE: 0.26 -> 0.41 | Helped |
| use adaptive layer-wise prefix KV retention instead of Elastic | LLaVA-1.5-13B · MM-Vet, 20% cache budget | PPL: 11.6 -> 5.16 | Helped |
| use adaptive layer-wise prefix KV retention instead of full cache | LLaVA-1.5-7B · batch size 16, token length 1024+512 | throughput (token/s): 266.6 -> 486.7 | Helped |
| allocate KV cache sizes adaptively across layers rather than retaining the same size in every layer (ablation) | PrefixKV · MM-Vet, LLaVA-1.5-7B, 30% cache budget | PPL: 14.7 PPL | Helped |
| estimate the global prefix configuration offline from random samples instead of online binary searching (ablation) | PrefixKV · MM-Vet | PPL | No clear effect |
| combine PrefixKV with positional cache merging (ablation) | PrefixKV · MM-Vet, LLaVA-1.5-7B | PPL | Hurt |
| combine PrefixKV with feature-similarity cache merging (ablation) | PrefixKV · MM-Vet, LLaVA-1.5-7B | PPL | Helped |
Surprising findings
- Layer-wise retained KV cache sizes show a W-shaped trend across layers, and layers with more concentrated importance (higher Gini) receive fewer cache entries.
Limitations
- Evaluated only on relatively small models (up to 13B) due to limited academic resources.
- Combination with distillation and pruning left for future work.
Experimental setup
Datasets: LLaVA-1.5 training set (10 random samples for offline estimation)
Benchmarks: LLaVA-Description MM-Vet LongBench Needle-in-a-HayStack
Metrics: PPL ROUGE-L F1 latency throughput mean absolute error
Baselines: Local (StreamingLLM) H2O PyramidKV Elastic Cache KIVI
Builds on: H2O PyramidKV Elastic Cache StreamingLLM FastV
Closest papers by meaning
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.92)
- TGV-KV: Text-Grounded KV Eviction for Vision-Language Models (ICML 2026 · similarity 0.91)
- Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models (ICLR 2026 · similarity 0.90)
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · similarity 0.89)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.