Research map / Architectures and efficiency / Quantization
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MiniCache compresses KV caches across adjacent layers in the middle-to-deep depth of LLMs by SLERP-merging direction vectors while keeping per-token magnitudes and retaining highly distinct tokens, training-free and orthogonal to quantization.
Problem
KV cache memory grows linearly with sequence length and batch size, limiting long-context and batched LLM inference; prior compression works focus on intra-layer redundancy (quantization, token eviction) and ignore inter-layer redundancy.
Approach
Observes that KV states of adjacent middle-to-deep layers are highly similar; merges paired layers' KV caches via spherical linear interpolation of normalized direction vectors, stores per-token magnitudes to restore norms, and keeps a small set of highly dissimilar tokens unmerged.
Key results
- On LongBench with Llama-2-7B-Chat, MiniCache combined with 4-bit KIVI reaches a 5.02x compression ratio with average score 35.44 vs. 36.41 baseline.
- On ShareGPT with LLaMA-2-7B at batch size 128, 4-bit MiniCache reduces peak memory by 25GB (41%) and gives roughly 5x throughput over FP16 baseline, and 1.29x over 2-bit KIVI.
- On GSM8K with LLaMA-3-70B, merging KV caches across all layers with MiniCache keeps accuracy near 0.80 in early-to-middle layers, whereas simple averaging drops to about 0.55.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MiniCache instead of full cache | LLaMA-2-7B-Chat · LongBench | Average: 36.41 -> 35.44 | Hurt |
| Use MiniCache instead of KIVI-2 | LLaMA-2-7B-Chat · LongBench | Average: 31.51 -> 35.44 | Helped |
| Use MiniCache instead of full cache | LLaMA-2-13B-Chat · LongBench | Average: 32.71 -> 32.61 | No clear effect |
| Use MiniCache instead of full cache | Mistral-7B · LongBench | Average: 37.33 -> 35.75 | Hurt |
| Use MiniCache instead of full cache | Mistral-7B-Instruct · LongBench | Average: 48.32 -> 46.99 | Hurt |
| Use MiniCache instead of KIVI-2 | Mistral-7B-Instruct · LongBench | Average: 43.74 -> 46.99 | Helped |
| Add 4-bit MiniCache rather than use the FP16 full-cache baseline | LLaMA-2-7B on ShareGPT batch serving · ShareGPT | Throughput: approximately 5× | Helped |
| Add 4-bit MiniCache rather than use 2-bit KIVI | LLaMA-2-7B on ShareGPT batch serving · ShareGPT | Throughput: 1.29× higher throughput | Helped |
| Set token retention threshold γ to 0.05 instead of 0 (ablation) | MiniCache with LLaMA-2-7B · COQA | score: 0.603 -> 0.647 | Helped |
| Set token retention threshold γ to 0.05 instead of 0.01 (ablation) | MiniCache with LLaMA-2-7B · COQA | score: 0.620 -> 0.647 | Helped |
| Set token retention threshold γ to 0.05 instead of 0.02 (ablation) | MiniCache with LLaMA-2-7B · COQA | score: 0.630 -> 0.647 | Helped |
| Set token retention threshold γ to 0.1 instead of 0.05 (ablation) | MiniCache with LLaMA-2-7B · COQA | score: 0.647 -> 0.643 | Hurt |
Surprising findings
- Simple averaging of adjacent-layer KV caches causes large accuracy drops, while preserving magnitudes and interpolating directions keeps performance near lossless.
- Merging distinct tokens without retention degrades performance, so a small fraction of outlier token pairs must be kept unmerged.
Limitations
- SLERP only merges two vectors at a time, so multi-layer merging is not supported.
- Accuracy degrades when merging is applied to layers early in the model, so merging starts from middle depth.
- Interpolation hyperparameters t and gamma were tuned on specific models and benchmarks.
Experimental setup
Datasets: ShareGPT COQA GSM8K TruthfulQA LongBench COPA MathQA OpenBookQA PIQA RTE WinoGrande XSUM
Benchmarks: GSM8K COQA TruthfulQA LongBench ShareGPT COPA MathQA OpenBookQA PIQA RTE WinoGrande XSUM
Metrics: accuracy compression ratio peak memory decoding throughput ROUGE
Baselines: FP16 full cache RTN SmoothQuant KIVI-2 H2O average merging
Builds on: SLERP weight normalization KIVI DoRA
Closest papers by meaning
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.94)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.94)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.94)
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.93)
- LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.93)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.