Research map / Architectures and efficiency / Quantization
CommVQ: Commutative Vector Quantization for KV Cache Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CommVQ, a vector quantization method for KV cache compression whose key codebook is commutative with RoPE, enabling 1–2 bit KV caches for long-context LLMs with reduced decoding overhead.
Problem
The KV cache becomes a GPU memory bottleneck for long-context LLM inference, and existing scalar quantization methods severely degrade accuracy at 2-bit and 1-bit, while vector-quantization decoding adds large computational overhead.
Approach
Quantize each key/value vector as a whole using additive quantization with a learned encoder and codebook, and design the key codebook to commute with the RoPE rotation matrices so that query-codebook products can be precomputed and reused across tokens during attention.
Key results
- CommVQ-2 (2-bit) achieves 47.98 LongBench average vs. 48.05 FP16 and 47.62 KIVI-2 at 3 bits.
- CommVQ-1 (1.03 bit) reaches 44.94 LongBench average vs. 27.42 for VQLLM-1, and 66.57 GSM8K vs. 1.67 for VQLLM-1.
- The optimized commutative-codebook implementation gives 6.0x to 9.6x per-layer decoding speedup over the naive implementation at 8K to 128K context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 2-bit CommVQ instead of FP16 KV cache | LLaMA-3.1-8B-Instruct · LongBench | Average: 48.05 -> 47.98 | No clear effect |
| Use CommVQ-2 instead of KIVI-2 | LLaMA-3.1-8B-Instruct · LongBench | Average: 47.62 -> 47.98 | Helped |
| Use CommVQ-1 instead of VQLLM-1 | LLaMA-3.1-8B-Instruct · LongBench | Average: 27.42 -> 44.94 | Helped |
| Use CommVQ-2 instead of FP16 KV cache | LLaMA-3.1-8B-Instruct · InfiniteBench | Average: 48.67 -> 42.75 | Hurt |
| Use CommVQ-1 instead of VQLLM-1 | LLaMA-3.1-8B-Instruct · InfiniteBench | Average: 18.48 -> 36.76 | Helped |
| Use CommVQ-2 instead of FP16 KV cache | LLaMA-3.1-8B-Instruct · GSM8K | Exact match accuracy: 76.27 -> 76.04 | No clear effect |
| Use CommVQ-1 instead of VQLLM-1 | LLaMA-3.1-8B-Instruct · GSM8K | Exact match accuracy: 1.67 -> 66.57 | Helped |
| Increase g while holding average quantization bit constant (R=1) (ablation) | CommVQ key-cache quantization · FineWeb-Edu first-layer key cache | MSE: 0.2699 -> 0.0906 | Helped |
| Increase g and R while maintaining 1-bit quantization (ablation) | CommVQ key-cache quantization · FineWeb-Edu first-layer key cache | MSE: 0.0798 -> 0.0095 | Helped |
| Increase R while holding g and N_c' unchanged (ablation) | CommVQ key-cache quantization · FineWeb-Edu first-layer key cache | MSE | Helped |
| Use additive CommVQ rather than asymmetric quantization | LLaMA-3.1-8B-Instruct value-cache quantization · FineWeb-Edu first-layer value cache at 1 bit | MSE: 0.00380 -> 0.00027 | Helped |
| Use optimized commutative-codebook implementation instead of naive implementation | CommVQ inference · 8K context | Latency (ms): 2.4 -> 0.4 | Helped |
Surprising findings
- KIVI-1 and KVQuant-1 nearly collapse at 1-bit on LongBench and GSM8K, while the learned VQ approach retains most accuracy.
Limitations
- Codebooks and encoder are calibrated on general text (FineWeb-Edu), with only a perplexity-based check of domain shift.
- Requires an EM-based calibration step and a learned encoder per model, adding offline cost.
- Decoding overhead grows with the number of residual codebooks R (R=11 for 1-bit, R=21 for 2-bit).
Experimental setup
Datasets: FineWeb-Edu
Benchmarks: LongBench InfiniteBench GSM8K Needle-in-a-Haystack RepoBench-P
Metrics: average score accuracy Avg. bit perplexity latency MSE
Baselines: KIVI KVQuant VQLLM asymmetric quantization
Builds on: KIVI KVQuant VQLLM additive quantization product quantization Gumbel-softmax
Closest papers by meaning
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · similarity 0.92)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.91)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.90)
- KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.90)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.90)
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.