Research map / Architectures and efficiency / Quantization

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

ICML 2024 · Architectures and efficiency › Quantization · approach: KV cache quantization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes KIVI, a tuning-free asymmetric 2-bit KV cache quantization method (per-channel keys, per-token values, with a full-precision residual window) that cuts peak memory 2.6x and raises throughput 2.35-3.47x.

Problem

KV cache memory and loading costs become the bottleneck for batched LLM inference with long contexts, and prior KV quantization work rarely studied the element distribution needed to push below 4 bits.

Approach

Analysis shows key cache has fixed outlier channels (quantize per-channel) while value cache is best quantized per-token because attention sparsity confines error to important tokens; a grouped-plus-full-precision-residual split makes per-channel key quantization compatible with streaming decoding.

Key results

• 2bit KIVI keeps accuracy within about 2% on GSM8K for Llama and Mistral models and achieves near-16bit LongBench averages (e.g., Llama2-7B 44.27 vs 44.52).
• On a single A100 with ShareGPT-style workloads on Llama-2-7B, KIVI reduces peak memory 2.6x, enables up to 4x larger batch size, and gives 2.35x-3.47x throughput.
• Fake 2bit per-channel-key plus per-token-value quantization gives the best accuracy among 2bit configurations, while per-channel value quantization collapses (e.g., CoQA 2.80 for K-T, V-C on Llama-2-13B).

Measured effects reported in the paper

Surprising findings

• Per-channel value quantization collapses accuracy regardless of key quantization, even though value cache shows no clear outlier pattern, which is explained by attention sparsity.
• Falcon-7B (multi-query attention, a single KV head) needs 4bit KIVI to hold accuracy; 2bit causes a large drop.

Limitations

• Speedup is reported only for Llama-2-7B on one GPU and the quantization overhead is not fused into prior operations, which limits efficiency gains.
• Falcon-7B with multi-query attention shows notable accuracy drops at 2bit, so the method is less effective for already-compressed KV caches.
• Closed-ended tasks like MMLU were excluded because they involve only one decoding step.

Experimental setup

Datasets: CoQA TruthfulQA GSM8K LongBench ShareGPT

Benchmarks: CoQA TruthfulQA GSM8K LongBench Needle-in-a-Haystack

Metrics: exact match accuracy BLEU F1 ROUGE classification score similarity score peak memory throughput

Baselines: FP16 baseline per-token 4bit KV quantization SmoothQuant FlexGen H2O StreamingLLM

Builds on: round-to-nearest quantization Hugging Face Transformers vLLM

Closest papers by meaning

• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · similarity 0.93)
• KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.93)
• KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.92)
• ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification (NeurIPS 2024 · similarity 0.91)
• Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.91)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)