Research map / Architectures and efficiency / Attention and transformers
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QuantSpec, a self-speculative decoding method where the draft model uses 4-bit weights and a hierarchical 4-bit/8-bit quantized KV cache (sharing bits with the target) plus a double full-precision buffer, yielding up to ~2.5x speedup for long-context LLM inference.
Problem
Speculative decoding for long-context LLM inference is bottlenecked by KV cache loading, and existing self-speculative methods using sparse KV caches (StreamingLLM, SnapKV) lose accuracy, giving low draft acceptance rates and high memory use.
Approach
The draft model shares the target architecture but uses a hierarchical quantized KV cache where the upper 4 bits serve as the INT4 draft cache and upper+lower 4 bits reconstruct the INT8 target cache, so no separate draft copy is stored; a double full-precision buffer of size 2G keeps recent tokens unquantized to raise acceptance and avoid repeated quantization on rollback.
Key results
- QuantSpec reaches up to 2.49x speedup over autoregressive decoding at 128k context on LWM-Text-Chat-128k (Multi-LexSum), with acceptance rate 94.31%.
- Acceptance rates above 90% are maintained across most settings, e.g., 91.64% on InfiniteBench Sum at 16k for LWM-Text-Chat-128k.
- The INT4 attention kernel is about 2.88x faster than FP16 FlashAttention at 64k context (0.54 ms vs 3.07 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QuantSpec instead of SnapKV | Self-speculative decoding · PG19, 4k | speedup over autoregressive generation: 1.17 -> 1.35 | Helped |
| Use QuantSpec instead of SnapKV | Self-speculative decoding · PG19, 8k | speedup over autoregressive generation: 1.31 -> 1.44 | Helped |
| Use QuantSpec instead of StreamingLLM | Llama-2-7B-32K-Instruct decoding · Multi-LexSum, 8k | speedup over autoregressive generation: 1.27 -> 1.61 | Helped |
| Use QuantSpec instead of StreamingLLM | Llama-2-7B-32K-Instruct decoding · Multi-LexSum, 32k | speedup over autoregressive generation: 1.84 -> 2.08 | Helped |
| Use QuantSpec instead of SnapKV | LWM-Text-Chat-128k decoding · ∞Bench Sum, 16k | speedup over autoregressive generation: 1.50 -> 1.78 | Helped |
| Use QuantSpec instead of SnapKV | LWM-Text-Chat-128k decoding · ∞Bench Sum, 32k | speedup over autoregressive generation: 1.78 -> 1.99 | Helped |
| Use QuantSpec instead of StreamingLLM | LWM-Text-Chat-128k decoding · Multi-LexSum, 64k | speedup over autoregressive generation: 2.16 -> 2.23 | Helped |
| Use QuantSpec instead of sparse-KV baselines | LWM-Text-Chat-128k decoding · Multi-LexSum, 128k | peak GPU memory: OOM -> 61.22 | Helped |
| Use INT8 instead of FP16 KV cache | Llama-2-7B · WikiText-2 | perplexity: 6.4595 -> 6.4696 | No clear effect |
| Use INT8 instead of FP16 KV cache | Llama-2-7B · C4 | perplexity: 7.2617 -> 7.2620 | No clear effect |
| Use an INT4 attention kernel that loads the upper 4-bit KV cache instead of FP16 FlashAttention | Attention kernel · 64k context | latency: 3.07 ms -> 0.54 ms (2.88x) | Helped |
| Use an INT4 attention kernel that loads the upper 4-bit KV cache instead of FP16 FlashAttention | Attention kernel · 256k context | latency: 6.16 ms -> 2.15 ms (2.86x) | Helped |
Surprising findings
- Sparse-KV self-speculative baselines (SnapKV) drop to 55.55% acceptance on Multi-LexSum at 8k with Llama-2-7B-32K, far below QuantSpec's 91.23%.
- The optimal speculation length for QuantSpec is larger (4 to 6) than for sparse-KV baselines, which peak at gamma=1.
Limitations
- Evaluated only at batch size 1 on a single-node 8x RTX A6000 setup
- Combination with sparse KV methods is left for future work
- Speedups are compared mainly against sparse-KV self-speculative baselines rather than a broader set of methods
Experimental setup
Datasets: WikiText-2 C4 PG-19 InfiniteBench Sum Multi-LexSum
Benchmarks: PG-19 InfiniteBench Sum Multi-LexSum WikiText-2 C4
Metrics: acceptance rate speedup vs autoregressive peak GPU memory perplexity kernel latency
Baselines: StreamingLLM SnapKV autoregressive decoding FlashAttention (FP16)
Builds on: speculative decoding FlashAttention FlashDecoding KIVI
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.90)
- MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (ICLR 2025 · similarity 0.90)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.90)
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.90)
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.90)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.