Research map / Architectures and efficiency / Quantization

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

ICML 2025 · Architectures and efficiency › Quantization · approach: KV cache quantization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AQUA-KV, a one-shot KV-cache compression method that trains linear predictors from previous-layer keys/values and quantizes only the unpredictable residuals, achieving near-lossless 2-2.5 bit KV caching on Llama 3.x and Qwen 2.5.

Problem

KV caches for long contexts consume tens of gigabytes; existing quantization, pruning and merging methods lose significant accuracy at about 2 bits per value.

Approach

Exploit inter-layer and intra-layer (key-value) dependencies by training compact linear predictors that estimate each cache entry from previously reconstructed entries, then quantize only the residual, with sequential layer-by-layer calibration on reconstructed inputs.

Key results

• On Llama 3.2 3B at 2.16 bits, AQUA-KV (HIGGS) reaches WikiText-2 PPL 7.03 vs 6.98 uncompressed and LongBench average 44.26 vs 44.61 uncompressed.
• 2-bit AQUA-KV over HIGGS is roughly equivalent to the 3-bit HIGGS baseline on Llama 3.x and Qwen 2.5 models.
• AQUA-KV with H2O pruning keeping 20% of tokens retains near-baseline LongBench (38.43 vs 38.82 H2O only on Llama 3.2 3B).

Measured effects reported in the paper

Surprising findings

• Attention sink tokens need to be kept uncompressed (first 4 tokens), since poorly compressed sinks change predictor inputs and degrade results.
• Linear predictors are sufficient; MLP predictors give only marginal gains over linear ones despite larger size.

Limitations

• Predictors are fit with plain linear regression and not optimized end-to-end for model-level objectives.
• Inference is slightly slower than the backbone quantizer alone (about 3% in the reported tests).
• Non-Instruct models perform poorly on LongBench, so main LongBench results use Instruct variants only.

Experimental setup

Datasets: RedPajama WikiText-2 LongBench

Benchmarks: WikiText-2 LongBench

Metrics: perplexity LongBench average score explained variance ratio memory footprint in GiB

Baselines: Quanto KIVI KVQuant HIGGS KVSharer QuaRot H2O

Builds on: HIGGS KVQuant KIVI H2O KVSharer

Closest papers by meaning

• KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.94)
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.94)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.91)
• Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs (NeurIPS 2025 · similarity 0.91)
• KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.91)