Research map / Architectures and efficiency / Quantization
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
KVQuant compresses the LLM KV cache to 2-4 bits via per-channel pre-RoPE key quantization, sensitivity-weighted non-uniform datatypes, per-vector dense-and-sparse outlier isolation, and attention-sink-aware fp16 first token, enabling 10M-token context on an 8-GPU system.
Problem
KV cache memory dominates inference cost at long context lengths, and existing KV cache quantization methods lose substantial accuracy below 4 bits.
Approach
Combines per-channel Key quantization applied before RoPE, offline sensitivity-weighted k-means non-uniform datatypes (nuqX), per-vector dense-and-sparse outlier isolation, and keeping the first attention-sink token in fp16 to quantize KV activations accurately at ultra-low bit widths.
Key results
- 3-bit KVQuant-1% achieves under 0.1 perplexity degradation vs fp16 on Wikitext-2 for LLaMA-7B (5.75 vs 5.68), giving 4.8x cache compression.
- On LLaMA-2-7B-32K RULER, 3-bit KVQuant scores 53.65 vs KIVI 39.78 average at similar bit width (3.33 vs 3.05).
- Custom kernels give 1.2-1.6x Key and 1.3-1.7x Value matvec latency savings over fp16 on A6000 for LLaMA-2-7B-32K, and the method enables 1M context on a single A100-80GB and 10M context on an 8-GPU system for LLaMA-7B (nuq2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use per-channel quantization for Keys and per-token quantization for Values instead of per-token quantization for both (ablation) | 3-bit KV cache quantization · Wikitext-2 | perplexity: 10.87 -> 7.05; 3.82 perplexity improvement | Helped |
| Use per-channel quantization for Values instead of per-token quantization (ablation) | 3-bit KV cache quantization · Wikitext-2 | perplexity: 7.05 -> 223 | Hurt |
| Quantize Keys before RoPE rather than after RoPE (ablation) | 3-bit KVQuant · Wikitext-2 | perplexity: 7.05 -> 6.23; 0.82 perplexity improvement | Helped |
| Add sensitivity weighting to the k-means objective (ablation) | nuq3 KV cache quantization · Wikitext-2 | perplexity: 6.84 -> 6.01 | Helped |
| Account for per-channel scaling factors when deriving k-means signposts (ablation) | Fisher-weighted nuq3 · Wikitext-2 | perplexity: 6.01 -> 5.94 | Helped |
| Add per-vector outlier thresholds instead of using a single per-matrix threshold (ablation) | nuq3 KV cache quantization with 1% outliers · Wikitext-2 | perplexity: 5.85 -> 5.75 | Helped |
| Add attention sink-aware quantization by keeping the first token in fp16 (ablation) | KVQuant across bit widths and evaluated models · Wikitext-2 | perplexity: 0.25 and 0.13 PPL improvements for nuq2-1% on Llama-3-8B and Llama-3-70B | Helped |
| Use KVQuant-3bit-1% instead of KIVI-2-gs32-r128 | LLaMA-2-7B-32K · LongBench average | score: 30.04 -> 31.21 | Helped |
| Use KVQuant-3bit-1% instead of KIVI-2-gs32-r128 | LLaMA-2-7B-32K · RULER average | accuracy: 39.78 -> 53.65; 14% better score | Helped |
Surprising findings
- Per-channel quantization of Values performs worse than per-token quantization, contrary to the intuition that per-channel grouping helps both Keys and Values.
- Per-channel Key quantization with grouping-free offline calibration works without fine-grained grouping, unlike concurrent work that needs grouping plus fp16 residual.
Limitations
- Per-channel Value quantization underperforms, so Values must use per-token quantization with online outlier threshold computation (offloaded to CPU).
- Non-uniform datatypes require offline calibration; 2-bit perplexity degradation remains above 0.5 for some models without outliers.
- Context-length claims (1M/10M) are based on memory estimates rather than end-to-end serving measurements.
Experimental setup
Datasets: Wikitext-2 C4
Benchmarks: Wikitext-2 C4 Passkey retrieval LongBench RULER
Metrics: perplexity success rate accuracy latency (microseconds) KV cache size (GB)
Baselines: int4 nf4 ATOM FlexGen KIVI int3 nf3
Builds on: SqueezeLLM NormalFloat RoPE
Closest papers by meaning
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024 · similarity 0.93)
- KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.93)
- CommVQ: Commutative Vector Quantization for KV Cache Compression (ICML 2025 · similarity 0.92)
- NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache (NeurIPS 2025 · similarity 0.92)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.91)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.