Research map / Large language models / Reasoning and chain of thought
EPIC: Efficient Position-Independent Caching for Serving Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes position-independent context caching (PIC) as compile/link steps and proposes EPIC, a serving system with the LegoLink algorithm that recomputes only the first k tokens of each cached chunk to mitigate attention sinks.
Problem
Prefix-based context caching requires exact prefix matches, limiting KV reuse for immutable chunks (documents, few-shot examples) that appear after varying prefixes; existing PIC (CacheBlend) has O(N^2) recomputation and heavy dynamic-sparsity overhead.
Approach
Compile each immutable chunk independently with position IDs from zero, then at link time recompute only the first k (≤32) tokens of each chunk (static sparsity) so they stop acting as attention sinks; recomputation cost scales with number of chunks rather than prompt length squared.
Key results
- LegoLink-2 keeps accuracy drops within 0-7% while reducing TTFT by up to 300% versus CacheBlend-15 in single-request synchronous workloads.
- EPIC (LegoLink-16) achieves up to 8x TTFT reduction compared to CacheBlend-15 under asynchronous multi-request workloads.
- EPIC achieves up to 7x higher throughput than CacheBlend-15; CacheBlend-15 hits OOM at about 35,000 tokens while LegoLink-16 avoids OOM until 50,000 tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LegoLink instead of CacheBlend | PIC serving · Six datasets, synchronous workload | TTFT: up to 3× | Helped |
| Use LegoLink-16 instead of CacheBlend-15 | PIC serving · Asynchronous workload | TTFT: 8× | Helped |
| Use LegoLink-16 instead of CacheBlend-15 | PIC serving · Asynchronous workload | throughput: 7× | Helped |
| Use LegoLink-16 instead of CacheBlend-15 | PIC serving · Long context, 512-token chunks | maximum context length before OOM: approximately 35,000 tokens -> 50,000 tokens | Helped |
| Recompute 2 initial tokens per chunk rather than use CacheBlend-15 (ablation) | LegoLink · Six datasets, synchronous workload | TTFT: up to 300% | Helped |
| Recompute 2 initial tokens per chunk rather than use CacheBlend-15 (ablation) | LegoLink · Six datasets, synchronous workload | accuracy drop: 0 - 7% | No clear effect |
| Recompute 1% or 5% of tokens with CacheBlend rather than use FR (ablation) | CacheBlend · Six datasets, synchronous workload | accuracy: up to 80% | Hurt |
| Increase the number of recomputed initial tokens per chunk (ablation) | LegoLink · Six datasets, synchronous workload | accuracy | Helped |
| Skip link-step recomputation and prepend then discard four dummy tokens during compilation (ablation) | LegoLink-0 · Synchronous workload | accuracy | No clear effect |
| Use static selection of chunk-initial tokens rather than dynamic attention-based selection | LegoLink · CacheBlend-15 synchronous workloads | runtime overhead: 16.37% - 63.56% | Helped |
Surprising findings
- LegoLink-0, which prepends dummy tokens during compile and discards their KV to remove attention sinks with zero link-time recomputation, preserves accuracy well.
- CacheBlend-1 and CacheBlend-5 show up to 80% worse accuracy than full recomputation, while LegoLink with few recomputed tokens restores most accuracy.
Limitations
- Evaluated only on small open models (7-9B) and on a single A100-80GB GPU; quantized larger models not tested.
- Sparsity-based methods can produce unnecessarily lengthy outputs, lowering F1/Rouge-L scores.
- Accuracy drops up to 7% relative to full recomputation in some settings.
Experimental setup
Datasets: 2WikiMQA MuSiQue HotpotQA SAMSum MultiNews Needle in a Haystack
Benchmarks: LongBench 2WikiMQA LongBench MuSiQue LongBench HotpotQA LongBench SAMSum LongBench MultiNews Needle in a Haystack
Metrics: TTFT F1 score Rouge-L throughput
Baselines: CacheBlend Fully Recompute Naive KV reuse PromptCache
Builds on: CacheBlend vLLM StreamingLLM
Closest papers by meaning
- Training Free Exponential Context Extension via Cascading KV Cache (ICLR 2025 · similarity 0.89)
- ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · similarity 0.88)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.88)
- KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse (NeurIPS 2025 · similarity 0.88)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.88)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.