Research map / Architectures and efficiency / Quantization
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QJL, a data-oblivious KV cache key quantizer that applies a Gaussian JL transform followed by sign-bit quantization, paired with an unbiased asymmetric inner-product estimator, reaching 3 bits per FPN with no accuracy loss.
Problem
KV cache memory grows with sequence length, and existing KV cache quantizers add 1-2 bits per number from storing full-precision quantization constants (zero point and scale) per block.
Approach
Apply a random Gaussian JL transform to key embeddings and keep only the sign bits plus the key norm, while the query is JL-projected without quantization; this asymmetric estimator is unbiased for the inner product, so no per-group constants are needed. Outlier channels are handled by a separate quantizer copy.
Key results
- On LongBench with longchat-7b-v1.5-32k at 3 bits per FPN, QJL scores 21.83 on NarrativeQA, 29.44 on Qasper and 35.62 on HotpotQA, matching or beating FP16 and KIVI and KVQuant on those datasets.
- On LM-eval with Llama-2-7B, QJL at 3 bits matches the FP16 baseline within about 0.1 points on Lambada, HellaSwag, PIQA and MMLU.
- The paper claims a more than fivefold reduction in KV cache memory (about 81% reduction versus 16 bits).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3-bit QJL instead of FP16 | longchat-7b-v1.5-32k · LongBench NarrativeQA | F1: 20.79 -> 21.83 | Helped |
| Use 3-bit QJL instead of FP16 | longchat-7b-v1.5-32k · LongBench Qasper | F1: 29.42 -> 29.44 | No clear effect |
| Use 3-bit QJL instead of FP16 | longchat-7b-v1.5-32k · LongBench MultiQA-en | F1: 42.83 -> 41.52 | Hurt |
| Use 3-bit QJL instead of FP16 | longchat-7b-v1.5-32k · LongBench MultifQA-zh | F1: 34.33 -> 34.42 | No clear effect |
| Use 3-bit QJL instead of FP16 | longchat-7b-v1.5-32k · LongBench HotpotQA | F1: 33.05 -> 35.62 | Helped |
| Use 3-bit QJL instead of FP16 | longchat-7b-v1.5-32k · LongBench 2WikiMultiQA | F1: 24.14 -> 23.60 | Hurt |
| Use 3-bit QJL instead of FP16 | Llama-2-7B · LM-eval Lambada-OpenAI | accuracy: 73.90 -> 73.88 | No clear effect |
| Use 3-bit QJL instead of FP16 | Llama-2-7B · LM-eval HellaSwag | accuracy: 57.18 -> 57.14 | No clear effect |
| Use 3-bit QJL instead of FP16 | Llama-2-7B · LM-eval PIQA | accuracy: 78.07 -> 78.07 | No clear effect |
| Use 3-bit QJL instead of FP16 | Llama-2-7B · LM-eval MathQA | accuracy: 28.11 -> 28.17 | No clear effect |
| Use 3-bit QJL instead of FP16 | Llama-2-7B · LM-eval MMLU | accuracy: 41.85 -> 41.78 | No clear effect |
| Use 3-bit QJL instead of BF16 | Llama-3-8B · LM-eval PIQA | accuracy: 79.65 -> 79.87 | Helped |
Surprising findings
- Quantizing only one side of the JL projection to sign bits yields an unbiased inner-product estimator, whereas quantizing both sides gives a biased estimate of the cosine.
- The distortion constants of the sign-quantized asymmetric estimator are smaller than those of the unquantized JL transform.
Limitations
- Only keys are quantized to 1 bit via QJL; values use standard token-wise quantization.
- Full CUDA implementation of the complete pipeline is deferred to future work; the wrapper is PyTorch.
- Results on LongBench cover only six datasets and one long-context model, with several differences within noise.
Experimental setup
Datasets: LongBench LM-eval tasks (Lambada-OpenAI, HellaSwag, PIQA, MathQA, MMLU)
Benchmarks: LongBench LM-eval
Metrics: F1 accuracy wall-clock time memory usage
Baselines: KIVI KVQuant FP16 exact cache
Builds on: Johnson-Lindenstrauss transform KIVI KVQuant
Closest papers by meaning
- NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache (NeurIPS 2025 · similarity 0.90)
- CommVQ: Commutative Vector Quantization for KV Cache Compression (ICML 2025 · similarity 0.89)
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · similarity 0.89)
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024 · similarity 0.89)
- KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.89)
- ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.