Research map / Architectures and efficiency / Quantization
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NSNQuant, a calibration-free vector quantization method for low-bit KV cache compression that uses a Normalize-Shift-Normalize transform with Hadamard rotation to align KV distributions to a standard normal, enabling one reusable codebook.
Problem
KV cache size limits LLM long-context inference; existing calibration-based vector quantization (CQ) for KV cache degrades under distribution shift between calibration and test data.
Approach
A token-wise normalize, channel-wise centering, and second token-wise normalize (NSN) followed by a Hadamard transform makes key/value channels approximately standard normal, so a single codebook tuned on synthetic torch.randn data can quantize them without calibration data.
Key results
- NSNQuant-1b reaches 47.38 LongBench average on LLaMA3.1-8B-Instruct vs 40.78 for CQ-8c10b at 1.23 vs 1.27 bits.
- On C4 with LLaMA3-8B, NSNQuant-2b gets PPL 9.08 vs 13.97 for CQ-4c9b, which degrades under distribution shift.
- NSNQuant achieves about 4x larger batch sizes and up to 3x throughput over FP16 on a single A100-80GB with LLaMA2-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use calibration-free NSNQuant-2b instead of CQ-4c9b | KV cache quantization · C4, LLaMA3.1-8B | PPL: 12.24 -> 9.15 | Helped |
| Use calibration-free NSNQuant-1b instead of CQ-8c10b | KV cache quantization · C4, LLaMA3.1-8B | PPL: 49.60 -> 17.20 | Helped |
| Use NSNQuant-1b instead of CQ-8c10b | LLaMA3.1-8B-Instruct · LongBench Avg. | Avg.: 40.78 -> 47.38 | Helped |
| Use NSNQuant-2b instead of KIVI-2 + Had | Mistral-7B-Instruct-v0.3 · LongBench Avg. | Avg.: 52.94 -> 52.75 | No clear effect |
| remove first token-wise normalization (ablation) | NSNQuant-2b · WikiText-2, LLaMA2-7B | PPL: 6.293 -> 5.285 | Helped |
| add channel-wise centering (ablation) | NSNQuant-2b · WikiText-2, LLaMA2-7B | PPL: 5.842 -> 5.285 | Helped |
| add second token-wise normalization (ablation) | NSNQuant-2b · WikiText-2, LLaMA2-7B | PPL: 5.456 -> 5.285 | Helped |
| add Hadamard transform (ablation) | NSNQuant-2b · WikiText-2, LLaMA2-7B | PPL: 5.730 -> 5.285 | Helped |
| fine-tune the K-Means codebook on synthetic standard-normal data (ablation) | NSNQuant-1b · WikiText-2, LLaMA2-7B | PPL: 6.910 -> 6.703 | Helped |
| replace naive Hadamard transform with randomized Hadamard transform (ablation) | NSNQuant-2b · WikiText-2, LLaMA2-7B | PPL: 5.285 -> 5.290 | No clear effect |
| increase residual size from 64 to 128 (ablation) | NSNQuant-1b · GSM8K (8-shot, CoT), LLaMA3.1-8B-Instruct | accuracy: 53.45 -> 57.54 | Helped |
| replace strategy 3 scaling with no scaling (ablation) | NSNQuant-2b · WikiText-2, LLaMA2-7B | PPL: 5.395 -> 5.285 | Helped |
Surprising findings
- CQ's calibration-induced errors concentrate on punctuation tokens (e.g., ',' without whitespace) in LLaMA3 models, causing large C4 perplexity degradation.
- Naive Hadamard transform performs as well as randomized Hadamard transform in practice.
Limitations
- Standard normal alignment fails in early layers with outlier channels, where variance bounds break down.
- 1-bit performance is sensitive to residual size; NSN adds about 0.23 bits overhead and higher prefill latency.
- E8P codebook is hard to apply to 1-bit settings.
Experimental setup
Datasets: WikiText-2 C4 LongBench GSM8K HumanEval CoQA MMLU ShareGPT
Benchmarks: WikiText-2 C4 LongBench GSM8K HumanEval CoQA MMLU
Metrics: perplexity accuracy F1 ROUGE-L throughput peak memory latency
Baselines: KIVI-2 KIVI-2 + Had KVQuant-2b + 1% KVQuant-1b + 1% CQ-4c9b CQ-8c10b E8P
Builds on: Coupled Quantization (CQ) QuIP# KIVI KVQuant QLoRA
Closest papers by meaning
- KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.92)
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · similarity 0.92)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.90)
- QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead (AAAI 2025 · similarity 0.90)
- CommVQ: Commutative Vector Quantization for KV Cache Compression (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.