Research map / Architectures and efficiency / Attention and transformers
SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpeCache, a training-free KV cache method that offloads the full 16-bit KV cache to CPU and keeps a low-bit (1/2-bit) GPU copy, using a speculative token decoded in parallel to predict and prefetch top-k 16-bit KV pairs for the next step.
Problem
LLM decoding on long sequences is bottlenecked by KV cache memory in GPU VRAM; existing compression (eviction, merging, quantization) is irreversible and offloading incurs CPU-GPU transfer latency.
Approach
Keep a low-bit KV copy in VRAM and the full 16-bit cache in CPU RAM; decode a speculative token alongside the output token to predict next-step top-k attended KV pairs, prefetching them in parallel with computation. Also modifies 1-bit KIVI quantization (new zero-point and scale).
Key results
- With 1-bit KIVI plus SpeCache, Mistral-7B-Instruct-v0.2 averages 40.4 on LongBench at 0.10x KV size versus 42.3 for full cache and 28.9 for 1-bit KIVI alone.
- On A6000 with Mistral-7B at 32k context, 1-bit SpeCache raises decoding throughput to 47.3 from 10.3 tokens/s (4.6x) by increasing batch size from 3 to 36.
- LLaMA-3-8B-Instruct with 1-bit SpeCache averages 41.9 on LongBench at 0.11x KV size, versus 42.7 for full cache.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add speculative KV prefetch to 2-bit KIVI (g=32) | LLaMA-2-7B-Chat · LongBench average | average score: 28.1 -> 29.1 | Helped |
| Add speculative KV prefetch to 2-bit KIVI (g=64) | LLaMA-2-7B-Chat · LongBench average | average score: 27.5 -> 29.1 | Helped |
| Add speculative KV prefetch to 2-bit KIVI | LLaMA-2-13B-Chat · LongBench average | average score: 30.0 -> 30.7 | Helped |
| Add speculative KV prefetch to 2-bit KIVI | Mistral-7B-Instruct-v0.2 · LongBench average | average score: 40.7 -> 41.5 | Helped |
| Add speculative KV prefetch to 1-bit KIVI (g=32) | Mistral-7B-Instruct-v0.2 · LongBench average | average score: 30.4 -> 40.4 | Helped |
| Add speculative KV prefetch to 1-bit KIVI | LLaMA-3-8B-Instruct · LongBench average | average score: 26.7 -> 41.8 | Helped |
| Use improved zero-point and scaling factor for 1-bit KIVI (ablation) | LLaMA-3-8B-Instruct with SpeCache · LongBench Qasper | score: 4.1 -> 43.2 | Helped |
| Use improved zero-point and scaling factor for 1-bit KIVI (ablation) | LLaMA-3-8B-Instruct with SpeCache · LongBench 2WikiMQA | score: 6.5 -> 36.9 | Helped |
| Use improved zero-point and scaling factor for 1-bit KIVI (ablation) | LLaMA-3-8B-Instruct with SpeCache · LongBench RB-P | score: 21.6 -> 51.0 | Helped |
| Replace speculative-token prefetch with output-token-based non-speculative fetching (ablation) | SpeCache, 2-bit · LongBench average, 2k context | latency (ms/step): 643 -> 877 | Hurt |
| Replace speculative-token prefetch with output-token-based non-speculative fetching (ablation) | SpeCache, 1-bit · LongBench average, 2k context | latency (ms/step): 775 -> 1144 | Hurt |
| Increase the number of prefetched KV pairs k (ablation) | SpeCache · LongBench | model performance | Helped |
Surprising findings
- Speculative prefetching with a 1-bit KV copy raises LongBench averages by up to 17 points over 1-bit KIVI alone for LLaMA-3-8B-Instruct.
Limitations
- CPU-GPU sparse transfer is about 5x slower per transfer than full transfer, and implementation is not theoretically optimal (PyTorch multi-stream).
- 1-bit quantization is poor for LLaMA-2 so experiments on it were limited to 2-bit.
- Uses GPT-4 scoring for NIAH, and relies on KIVI-based quantization without evaluating other quantizers in depth.
Experimental setup
Datasets: PG19 LongBench Needle-in-a-Haystack
Benchmarks: LongBench Needle-in-a-Haystack
Metrics: F1/score on LongBench tasks NIAH score (1-10) decoding throughput decoding latency (ms/step)
Baselines: KIVI InfLLM StreamLLM H2O
Builds on: KIVI H2O FlexGen InfLLM
Closest papers by meaning
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.93)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.92)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.92)
- EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.91)
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.91)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.