Research map / Architectures and efficiency / Quantization
KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KVTuner, an offline search for layer-wise mixed key/value KV cache quantization precision pairs (e.g., K8V4, K4V2) using multi-objective optimization with search-space pruning, achieving near-lossless ~3.25–4-bit KV cache on reasoning tasks.
Problem
Low-bit KV cache quantization degrades LLM accuracy, especially in sensitive layers and for certain models (e.g., Qwen2.5-7B-Instruct), while fine-grained online mixed-precision methods add overhead and are hard to deploy on static-graph/flash-attention kernels.
Approach
Quantize key and value caches of each layer at different precisions, with keys prioritized, and search per-layer precision pairs offline via multi-objective optimization (Optuna/MOEA-D), using intra-layer Pareto pruning and inter-layer clustering to shrink the search space; the resulting static configuration adds no online overhead.
Key results
- KVTuner-C3.25 on Llama-3.1-8B-Instruct with KIVI keeps GSM8K 4-shot average accuracy at 0.7925 vs. 0.8033 for KIVI-8 at 3.25-bit equivalent precision.
- On Qwen2.5-7B-Instruct with per-token-asym, KVTuner-C4.0 reaches 0.7559 average GSM8K accuracy while uniform KV4 collapses to 0.0078.
- KVTuner-C3.25 improves Llama-3.1-8B-Instruct decoding throughput by 16.79%–21.25% over KIVI-KV8 across batch size/input length settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use layer-wise mixed-precision KV quantization instead of uniform KV8 | KVTuner with KIVI · Llama-3.1-8B-Instruct throughput, batch size 64 and input length 128 | tokens/s: 3836 -> 4652 (+21.25%) | Helped |
| Use layer-wise mixed-precision KV quantization at 3.25-bit | Llama-3.1-8B-Instruct · GSM8K few-shot CoT and multiturn average | accuracy: 0.8033 -> 0.7925 | No clear effect |
| Use KVTuner mixed precision instead of uniform KV4 | Qwen2.5-7B-Instruct · LongBench 20-dataset average | score: 0.6343 -> 0.7960 | Helped |
| Reduce key precision from 4-bit to 2-bit in a sensitive layer group | Llama-3.1-8B-Instruct · GSM8K calibration | accuracy: 0.67 -> 0.495 | Hurt |
| Prioritize key precision over value precision at matched memory use | KV cache quantization · Llama-3.1-8B-Instruct, GSM8K | relative attention output error: K4V2: 0.453; K2V4: 0.892 | Helped |
| Use per-channel-asym rather than per-token-asym quantization for keys | Llama-3.1-8B-Instruct · GSM8K zero-shot calibration | relative key error: 0.012280 -> 0.004869 | Helped |
| Reduce uniform KV precision from 8-bit to 4-bit | Qwen2.5-7B-Instruct with per-token-asym quantization · GSM8K few-shot CoT and multiturn average | accuracy: 0.7712 -> 0.0078 | Hurt |
| Disable intra-layer and inter-layer search-space pruning (ablation) | KVTuner search · Llama-3.1-8B-Instruct, GSM8K 4-shot | search convergence and quantization performance | Hurt |
| Use KIVI-2 instead of KIVI-8 | Llama-3.1-8B-Instruct · GSM8K few-shot CoT and multiturn average | accuracy: 0.8033 -> 0.6327 | Hurt |
Surprising findings
- Uniform KV4 quantization collapses Qwen2.5-7B-Instruct to near 0% on GSM8K while K8V2-style mixed precision stays near lossless.
- Longer chain-of-thought with lower mixed-precision KV cache can outperform short CoT with BF16 KV cache on mathematical reasoning.
Limitations
- Search is performed offline per model and quantization mode, requiring calibration and several hundred Optuna iterations per setting.
- Throughput results are reported only for Llama-3.1-8B-Instruct and rely on the KIVI GPU kernel; implementation is in huggingface transformers.
Experimental setup
Datasets: GSM8K (first 200 4-shot prompts for search calibration) first 20 zero-shot GSM8K prompts for intra-layer pruning wikitext
Benchmarks: GSM8K GPQA Extended LongBench CEVAL MMLU TriviaQA RACE TruthfulQA wikitext word perplexity
Metrics: accuracy word perplexity equivalent precision (bits) decoding throughput (tokens/s)
Baselines: KIVI-8 KIVI-4 KIVI-2 per-token-asym KV8 per-token-asym KV4 per-token-asym KV2 K8V4 K4V2
Builds on: KIVI Optuna MOEA/D lm-evaluation-harness
Closest papers by meaning
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.93)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.93)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.93)
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · similarity 0.93)
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024 · similarity 0.92)
- NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.