Research map / Architectures and efficiency / Quantization
KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Coupled Quantization (CQ), which jointly quantizes groups of contiguous KV cache channels with Fisher-guided multi-channel centroids, to compress LLM KV cache down to 1 bit per floating-point number.
Problem
KV cache size and memory bandwidth dominate LLM inference cost at long context or large batch, and existing per-channel or per-token quantization degrades sharply at low bit widths (2 bits and below).
Approach
Exploit the strong inter-channel dependency of key/value activations: group contiguous channels and quantize each group jointly with a shared multi-dimensional codebook learned by k-means, optionally weighted by Fisher information, so joint entropy is lower than the sum of marginal entropies.
Key results
- On WikiText-2, CQ-8c8b (1 bit per FPN) reaches perplexity 8.09 on LLaMA-7b versus 321.58 for KVQuant-1b, while KVQuant-1b-1% reaches 9.93.
- CQ-4c8b (2 bits per FPN) matches or beats KVQuant-2b-1% on several models, e.g., LLaMA-2-13b perplexity 4.81 vs 4.92 on WikiText-2.
- CQ-8c10b (1.25 bits per FPN) reaches 6.78 perplexity on LLaMA-7b WikiText-2, outperforming all 1-bit baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use channel-coupled quantization instead of 1-bit KVQuant without sparse outliers | LLaMA-7b · WikiText-2 test | perplexity: 321.58 -> 8.09 | Helped |
| Use channel-coupled quantization instead of 2-bit KVQuant without sparse outliers | LLaMA-7b · WikiText-2 test | perplexity: 8.17 -> 5.97 | Helped |
| Use channel-coupled quantization instead of 1-bit KVQuant without sparse outliers | LLaMA-7b · C4 test | perplexity: 168.90 -> 12.13 | Helped |
| Use channel-coupled quantization instead of 2-bit KVQuant without sparse outliers | LLaMA-7b · C4 test | perplexity: 10.28 -> 7.52 | Helped |
| Use channel-coupled quantization instead of sparse-outlier KVQuant at 1 bit | LLaMA-7b · C4 test | perplexity: 11.18 -> 12.13 | Hurt |
| Use channel-coupled quantization instead of sparse-outlier KVQuant at 2 bits | LLaMA-7b · C4 test | perplexity: 7.38 -> 7.11 | Helped |
| Increase channel coupling from 1 to 4 channels (ablation) | CQ with uniform centroids on Mistral-7b · WikiText-2 test | perplexity: 97.76 -> 5.42 | Helped |
| Increase channel coupling from 1 to 4 channels (ablation) | CQ with Fisher-guided centroids on Mistral-7b · WikiText-2 test | perplexity: 5.34 -> 5.11 | Helped |
| Add Fisher-guided centroid learning (ablation) | CQ with 1 coupled channel on Mistral-7b · WikiText-2 test | perplexity: 97.76 -> 5.34 | Helped |
| Add Fisher-guided centroid learning (ablation) | CQ with 4 coupled channels on Mistral-7b · WikiText-2 test | perplexity: 5.42 -> 5.11 | Helped |
| Increase channel coupling from 1 to 4 channels (ablation) | CQ with uniform centroids on LLaMA-2-13b · WikiText-2 test | perplexity: 890.42 -> 6.62 | Helped |
| Add Fisher-guided centroid learning (ablation) | CQ with 1 coupled channel on LLaMA-2-13b · WikiText-2 test | perplexity: 890.42 -> 6.06 | Helped |
Surprising findings
- Plain INT2 quantization without grouping collapses to perplexities in the thousands on several models, showing how fragile low-bit KV quantization is.
Limitations
- Perplexity at 1 bit remains far from FP16 (e.g., 8.09 vs 5.68 on LLaMA-7b), so 1-bit quality is only partially preserved.
- Centroid learning requires an offline calibration step per model and configuration.
- Evaluation limited to five models up to 13B parameters and perplexity plus three zero-shot tasks.
Experimental setup
Datasets: WikiText-2 C4
Benchmarks: WikiText-2 C4 WinoGrande PIQA ARC Challenge
Metrics: perplexity accuracy quantization error bits per floating-point number
Baselines: INT quantization NormalFloat (NF) quantization KVQuant KIVI
Builds on: KVQuant KIVI Product Quantization SqueezeLLM
Closest papers by meaning
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.94)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.94)
- KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.93)
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024 · similarity 0.93)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
- CommVQ: Commutative Vector Quantization for KV Cache Compression (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.