Research map / Large language models / Reasoning and chain of thought
Compute or Load KV Cache? Why Not Both?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cake, a KV cache loader that bidirectionally computes early chunks on GPU while loading later chunks from storage in parallel, plus an adaptive scheduler that shares compute with non-prefix requests.
Problem
Prefix caching reduces prefill compute but loading cached KV caches from low-bandwidth disk or network storage adds large TTFT latency for long-context inputs.
Approach
Compute KV cache from the start of the sequence while concurrently loading it from storage starting at the end, exploiting that compute cost grows with token position while I/O cost is uniform; the two meet in the middle, and the merge point adapts to resource fluctuations.
Key results
- Cake achieves on average 2.6x TTFT reduction versus compute-only and I/O-only methods (abstract).
- Across hardware configs, Cake averages 2.23x over I/O-only and 3.76x over compute-only after excluding extreme outliers.
- Adaptive scheduling reduces overall finish time from 1.5s to 1.19s, improving throughput by 26%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine bidirectional KV-cache computation and I/O loading | Long-context prefix-cache loading · Long-context prefix-caching scenarios | TTFT reduction: 2.6× | Helped |
| use adaptive scheduling to prioritize decoding and non-prefix-cache prefill | Cake · 16K prefix-caching request with 22 additional requests | throughput: from 1.5s to 1.19s; 26% | Helped |
| increase sequence length | Cake · LongAlpaca-13B, fixed compute and I/O configuration | speedup over compute-only | Helped |
| use Cake on a short sequence when I/O bandwidth is 7 Gbps | Cake · LongAlpaca-13B, 4k tokens, 2xA100, 100% utilization | speedup over compute-only: 0.99 | Hurt |
| use GQA instead of MHA | Cake · LLaMA 3.1-8B versus LongAlpaca-7B, same compute and I/O settings | speedup over compute-only | Helped |
| increase model size within the same model series | Cake · LongAlpaca and LLaMA 3.1 model series | speedup over I/O-only | Hurt |
| reduce KV-cache precision from 16-bit to 8-bit or 3-bit | Cake · LongAlpaca-13B, 2xA100, 100% utilization, 32 Gbps | speedup over compute-only: 1.59; 2.12; 3.73 | Helped |
| assign some work to I/O even when computation is significantly faster | Cake · Short sequences with 2×A100 and 7 Gbps bandwidth | speedup over compute-only | Hurt |
Surprising findings
- Cake can underperform compute-only when sequences are short and compute is much faster than a 7 Gbps link, because the scheduler still assigns some work to I/O.
Limitations
- I/O bandwidth and storage were simulated via injected delays rather than real storage.
- Scheduler lacks an estimation mechanism, so it can add overhead when one resource dominates.
- Only prefix-cached KV caches precomputed in advance are evaluated; evaluation uses synthetic prompt lengths.
Experimental setup
Datasets: LongChat TriviaQA NarrativeQA
Benchmarks: LongChat TriviaQA NarrativeQA
Metrics: time-to-first-token (TTFT) speedup over I/O-only and compute-only throughput
Baselines: vLLM chunk prefill (compute-only) LMCache (I/O-only) CacheGen
Builds on: vLLM LMCache chunked prefill CacheGen
Closest papers by meaning
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.91)
- Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective (ICLR 2026 · similarity 0.89)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.89)
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.89)
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.