Research map / Architectures and efficiency / Attention and transformers
MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MPCache, a training-free, MPC-friendly KV cache eviction framework combining look-once static eviction with query-aware hierarchical clustered dynamic selection, cutting private LLM decoding latency and communication.
Problem
Private LLM inference via secure multi-party computation (MPC) suffers high latency and communication for long sequences, mainly from attention; existing plaintext KV cache eviction methods use MPC-unfriendly operations (top-k ranking, token gathering, similarity), so they do not directly help.
Approach
Combine one-time static eviction of unimportant tokens during prefill with dynamic per-step selection over clusters of KV cache, using an MPC-friendly upper-bound similarity (max/min key bounds) linearized into a weighted sum (alpha=0.6) to avoid max operations, hierarchical coarse-to-fine cluster selection, and sharing top-k indices across adjacent layers.
Key results
- On LongBench with LongChat-7B, MPCache matches full-cache F1 on HotpotQA and NarrativeQA at about 5% KV budget while reducing latency 2.24x vs LongCache.
- In 3PC efficiency evaluation with sequence lengths 512-2048 and 70% static eviction, MPCache achieves 1.26-2.01x latency reduction and 3.39-8.37x communication reduction versus full KV cache.
- Ablation on LLaMA-2-7B (length 1024): MPCache reaches 1.9x latency and 5.9x communication reduction versus the dynamic-selection baseline with static eviction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MPCache instead of InfLLM | LongChat-7B-V1.5-32K · HotpotQA, 5% cache budget | performance (%): 28.20 -> 30.27 | Helped |
| use MPCache instead of InfLLM | LongChat-7B-V1.5-32K · HotpotQA, 5% cache budget | latency (s): 51.64 -> 39.85 | Helped |
| use MPCache instead of InfLLM | LongChat-7B-V1.5-32K · TriviaQA, 5% cache budget | performance (%): 75.65 -> 75.61 | No clear effect |
| use MPCache instead of InfLLM | LongChat-7B-V1.5-32K · NarrQA, 5% cache budget | performance (%): 12.80 -> 17.23 | Helped |
| use MPCache instead of H2O | LLaMA-2-7B · XSUM, 7B, 5% budget | score: 4.886 -> 10.08 | Helped |
| use MPCache instead of SnapKV | LLaMA-3.1-8B-Instruct · LongBench NarrativeQA, budget 1024 | score: 26.51 -> 29.47 | Helped |
| use static eviction before dynamic selection (ablation) | dynamic KV cache selection · LLaMA-2-7B, sequence length 1024, static eviction ratio 70% | latency: 1.42× | Helped |
| use static eviction before dynamic selection (ablation) | dynamic KV cache selection · LLaMA-2-7B, sequence length 1024, static eviction ratio 70% | communication: 2.76× | Helped |
| add MPC-friendly clustering, linearization, reordering, and cross-layer index-sharing (ablation) | MPCache with static eviction and dynamic selection · LLaMA-2-7B, sequence length 1024, static eviction ratio 70% | latency and communication: 1.9× and 5.9× latency and communication reduction | Helped |
| apply dynamic token selection without static eviction or MPC-friendly optimizations (ablation) | private LLM inference · LLaMA-2-7B, sequence length 1024 | latency and communication | Hurt |
| increase hierarchical cluster size or decrease the coarse-grained selection ratio (ablation) | MPCache hierarchical KV cache selection · LongBench, dynamic selection ratio 20% | communication: 163.5 -> 69.12 MB | Helped |
| increase hierarchical cluster size or decrease the coarse-grained selection ratio (ablation) | MPCache hierarchical KV cache selection · LongBench, dynamic selection ratio 20% | F1 score: 29.6 -> 29.0 | Hurt |
Surprising findings
- Directly applying dynamic KV selection (LongCache) in MPC does not improve and can even increase latency and communication.
- Average-based key aggregation for cluster similarity degrades performance substantially compared with the max-bound approximation.
Limitations
- Static eviction may fail on tasks requiring recall of tokens deemed unimportant during prefill.
- MPC inference still incurs substantial overhead relative to plaintext inference.
- Efficiency evaluated with reduced embedding dimension (1024) rather than full 7B model size; alpha and hierarchy hyperparameters chosen empirically.
Experimental setup
Datasets: LongBench XSUM Needle-in-a-Haystack
Benchmarks: LongBench XSUM Needle-in-a-Haystack
Metrics: F1 score ROUGE decoding latency (s) communication (MB)
Baselines: StreamingLLM H2O TOVA SnapKV InfLLM LongCache DuoAttention Quest ArkVale head merging
Builds on: H2O Quest SnapKV Secretflow PUMA BumbleBee
Closest papers by meaning
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.89)
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.89)
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction (NeurIPS 2026 · similarity 0.88)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.88)
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.88)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.