Research map / Large language models / Reasoning and chain of thought
KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
KVLink precomputes per-document KV caches independently, re-encodes positions at inference, and inserts trainable link tokens to restore cross-document attention, enabling fast context reuse for RAG-style inputs.
Problem
Reusing precomputed KV caches for retrieved documents across queries is efficient, but independently encoded documents lose cross-document attention and position consistency, causing large accuracy drops in RAG and multi-segment tasks.
Approach
Decouple RoPE from stored KV caches and re-apply global positions at inference; append trainable link tokens to each independently encoded document whose KV states are computed at inference and attend to all preceding documents, restoring cross-segment self-attention with minimal overhead.
Key results
- KVLink5 on Llama-3.2-3B reaches 64.4% NQ and 69.5% HotpotQA, close to the fine-tuned full-context upper bound (69.7% and 74.3%) and above BlockAttention (58.8% and 64.3%).
- KVLink reduces time-to-first-token by 85%-96% relative to standard decoding; about 96% at 5,000-token context on Llama-3.1-8B.
- With 75% compression, KVLink improves NQ accuracy of the modified AnLLM compression over BlockAttention (40.9 vs 37.5 on Llama-3.2-1B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KVLink5 instead of BlockAttention | Llama-3.2-1B · HotpotQA | accuracy: 48.3% -> 55.6% | Helped |
| Add 5 link tokens instead of using KVLink0 (ablation) | KVLink with Llama-3.2-1B · HotpotQA | accuracy: 48.1% -> 55.6% | Helped |
| Add 5 link tokens instead of using KVLink0 (ablation) | KVLink with Llama-3.2-3B · MuSiQue | accuracy: 28.8% -> 35.8% | Helped |
| Omit summarization training data (ablation) | KVLink5 · Samsum | RougeL: 0.256 -> 0.250 | Hurt |
| Omit summarization training data (ablation) | KVLink5 · MuSiQue | accuracy: 19.2 -> 18.5 | Hurt |
| Omit multi-turn conversation training data (ablation) | KVLink5 · MuSiQue | accuracy: 19.2 -> 17.8 | Hurt |
| Omit multi-turn conversation training data (ablation) | KVLink5 · Samsum | RougeL: 0.256 -> 0.262 | Helped |
| Train on QA data only (ablation) | KVLink5 · MuSiQue | accuracy: 19.2 -> 17.3 | Hurt |
| Train on QA data only (ablation) | KVLink5 · NQ | accuracy: 45.0 -> 48.9 | Helped |
| Use KVLink5 instead of BlockAttention with modified AnLLMs compression | Llama-3.2-1B · NQ, 75% compression | accuracy: 37.5 -> 40.9 | Helped |
| Reuse precomputed KV cache with KVLink instead of full-context standard decoding | Llama-3.1-8B · 5,000-token context | TTFT reduction: 96% | Helped |
Surprising findings
- Retriever-ordered QA training data causes models to learn to find answers near the start of context, so shuffling retrieved documents is necessary.
Limitations
- Performance still degrades after KV cache compression.
- KV cache sizes vary with document length, making storage and vector-database organization difficult.
- Minor drops on some general benchmarks (e.g., ARC-C, Winogrande) relative to original Llama.
Experimental setup
Datasets: TriviaQA 2WikiMQA FineWeb Tulu 3 SFT mixture DaringAnteater XSum NaturalQuestions HotpotQA MuSiQue MultiNews SamSum IFEval
Benchmarks: NaturalQuestions 2WikiMQA TriviaQA HotpotQA MuSiQue MultiNews SamSum IFEval GSM8K MMLU ARC-Challenge ARC-Easy
Metrics: accuracy RougeL time-to-first-token
Baselines: PromptCache CacheBlend BlockAttention TurboRAG LLMLingua
Builds on: RoPE BlockAttention CacheBlend PromptCache AnLLM
Closest papers by meaning
- KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (NeurIPS 2025 · similarity 0.93)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.92)
- Training Free Exponential Context Extension via Cascading KV Cache (ICLR 2025 · similarity 0.91)
- KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs (NeurIPS 2026 · similarity 0.91)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.91)
- PatchKV: Weight Space Compensation of KV Cache (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.