Research map / Architectures and efficiency / Attention and transformers

KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes KV Packet, which wraps frozen document KV caches with trainable header/trailer soft-token adapters distilled from full-attention outputs, enabling recomputation-free concatenation of cached documents.

Problem

Reusing precomputed document KV caches in new multi-document contexts degrades quality because cached states are context-dependent; existing fixes need fine-tuning or costly selective recomputation at inference time, raising TTFT and FLOPs.

Approach

Treat each document cache as an immutable packet bounded by learned header and trailer soft-token adapters, trained by self-distillation so the model's full-attention output distribution is matched when packets are simply concatenated; no base-model weights change and no recomputation happens at serving time.

Key results

• KV Packet reduces FLOPs by 5-6 orders of magnitude versus Full Recompute, matching No Recompute (about 6.5e-6 to 1.04e-5).
• On Llama-3.1-8B, KV Packet gives 19.45x TTFT reduction on Needle-in-a-Haystack and 5.81x on MusiQue, with F1 comparable to full recomputation.
• Universal adapter trained on a mixture reaches F1 0.95 on Biography, 0.42 on HotpotQA, 0.43 on MusiQue, versus 0.01-0.24 for No Recompute.

Surprising findings

• Adapters trained on one dataset transfer poorly to others (e.g., Biography-trained adapter gets 0.18 on HotpotQA), so universal mixed training is needed.

Limitations

• Generalization to out-of-distribution corpora is untested.
• Only a limited set of model families evaluated.
• Targets independent documents; dependent multi-step reasoning chains not studied.

Experimental setup

Datasets: Needle-in-a-Haystack Biography HotpotQA MusiQue

Benchmarks: Needle-in-a-Haystack Biography HotpotQA MusiQue

Metrics: F1 FLOPs TTFT

Baselines: CacheBlend EPIC A3 SAM-KV Random recomputation No Recompute Full Recompute No Cache

Builds on: RoPE prompt tuning Block-Attention CacheBlend EPIC

Closest papers by meaning

• KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse (NeurIPS 2025 · similarity 0.91)
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (NeurIPS 2025 · similarity 0.91)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.90)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.90)
• Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective (ICLR 2026 · similarity 0.90)
• PatchKV: Weight Space Compensation of KV Cache (NeurIPS 2026 · similarity 0.89)