Research map / Architectures and efficiency / Attention and transformers
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ShadowKV, a long-context LLM inference system that stores low-rank pre-RoPE keys and offloads values to CPU, using chunk-level landmarks and static outlier chunks for accurate sparse decoding, achieving up to 3.04x throughput gain on A100.
Problem
High-throughput long-context LLM inference is limited by the growing KV cache, which consumes GPU memory and must be fetched during decoding; existing eviction and sparse attention methods lose accuracy, fail to reduce memory, or incur CPU-fetch latency.
Approach
Pre-RoPE keys are exceptionally low-rank, so the system keeps a rank-160 SVD projection of keys on GPU, offloads values to CPU, reconstructs selected keys on the fly, and selects chunks via mean-key landmarks plus a small set of GPU-resident outlier chunks.
Key results
- On RULER 128K with Llama-3.1-8B, ShadowKV reaches 83.57 average vs 85.53 full attention and 76.29 for Quest at a 1.56% sparse budget.
- ShadowKV supports up to 6x larger batch sizes and improves throughput up to 3.04x over full attention on A100 for Llama-3.1-8B at 122K context.
- Pre-RoPE keys can be compressed by over 6x (rank 160) without accuracy drop; theoretical equivalent bandwidth estimated at 7.2 TB/s on A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ShadowKV instead of Quest (V) | Llama-3-8B-1M · RULER 128K (Avg.) | accuracy: 83.99 -> 86.88 | Helped |
| Use ShadowKV instead of Quest (V) | GLM-4-9B-1M · LongBench >4K tokens (Avg.) | score: 43.68 -> 47.89 | Helped |
| Use ShadowKV instead of full attention | Llama-3.1-8B · Generation, 122K context | throughput (tokens/s): 80.78 -> 245.90; 3.04× | Helped |
| Add MInference prefill to ShadowKV (ablation) | Llama-3-8B-1M · RULER with MInference, Avg. | accuracy: 81.98 -> 82.04 | No clear effect |
| Add outlier chunks rather than using 0 (ablation) | ShadowKV on Llama-3-8B-1M · RULER 128K (Avg.) | accuracy: 74.05 -> 86.88 | Helped |
| Increase outlier chunks from 1 to 48 (ablation) | ShadowKV on Llama-3-8B-1M · RULER 128K (Avg.) | accuracy: 85.01 -> 86.88 | Helped |
| Increase chunk size beyond eight (ablation) | ShadowKV · long-context tasks | accuracy | Hurt |
| Increase pre-RoPE key rank up to approximately 160 (ablation) | ShadowKV · long-context tasks | accuracy | Helped |
| Increase pre-RoPE key rank beyond approximately 160 (ablation) | ShadowKV · long-context tasks | accuracy | No clear effect |
| Project generated-token keys into the prefill low-rank space (ablation) | ShadowKV on Llama-3-8B-1M · RULER 128K (Avg.) | accuracy: 86.88 -> 86.23 | Hurt |
| Project generated-token keys into the prefill low-rank space (ablation) | ShadowKV · LongBench >4K tokens | score: 39.94 -> 40.36; 48.24 -> 48.11; 48.96 -> 47.97; 36.85 -> 36.96 | Mixed |
Surprising findings
- ShadowKV sometimes exceeds full attention on specific RULER tasks such as variable tracking.
- Mean-based landmarks without outlier handling can underperform Quest's min-max approach, per the outlier ablation.
Limitations
- Eviction-free but still relies on CPU offloading of values, so fetch latency remains a bottleneck for very large KV.
- Generated tokens were initially kept in full precision; low-rank handling of new tokens is only an extension (ShadowKV+).
- Mean-based landmarks depend on outlier handling and may underperform min-max landmarks when outliers are not managed.
Experimental setup
Datasets: PG-19
Benchmarks: RULER LongBench Needle In A Haystack InfiniteBench
Metrics: accuracy throughput (tokens/s) latency (ms) memory savings
Baselines: Quest Loki InfiniGen Full Attention SnapKV StreamingLLM
Builds on: Quest InfiniGen Loki StreamingLLM MInference
Closest papers by meaning
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.93)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.91)
- SnapKV: LLM Knows What You are Looking for Before Generation (NeurIPS 2024 · similarity 0.91)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.91)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.