Research map / Architectures and efficiency / Attention and transformers
ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
ArkVale is a training-free, page-based KV cache manager that backs up filled KV pages to CPU, summarizes keys with bounding volumes, and recalls previously evicted pages whose estimated importance rises.
Problem
Long-context LLM generative inference is limited by KV cache memory and attention memory traffic; existing eviction methods permanently discard tokens that may become important later.
Approach
Organize KV cache into pages, asynchronously back up filled pages to CPU, keep a bounding-volume digest (sphere or cuboid) of keys on GPU to estimate each page's max query-key dot product, and dynamically recall/evict pages so top-k pages participate in attention.
Key results
- On LongBench with LongChat-7b-v1.5-32k, ArkVale nearly matches the full-cache origin at budgets above 2048 and outperforms StreamingLLM, H2O and TOVA across datasets and budgets.
- Passkey retrieval accuracy stays above 95% for ArkVale across 10k-30k contexts and 512-4096 budgets, while baselines fall below 50%.
- Decoding latency improves up to 2.2x (1.7x average) and batching throughput up to 4.6x (3.5x average) versus the full KV cache.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use recallable page eviction instead of permanent token eviction | ARKVALE vs. StreamingLLM, H2O, and TOVA · Passkey retrieval (10k, 20k, and 30k contexts; cache budgets 512–4096) | accuracy: above 95% | Helped |
| Use recallable page eviction rather than full KV cache | ARKVALE · Long-context decoding | latency: up to 2.2× speedup (1.7× in average) | Helped |
| Use recallable page eviction rather than full KV cache | ARKVALE · Long-context serving | throughput: up to 4.6× (3.5× in average) | Helped |
| Use page size 16 instead of page size 32 (ablation) | ARKVALE · LongBench tasks, especially tight cache budgets | F1 / Rouge-L / accuracy | Mixed |
| Estimate page importance with cuboid digests instead of sphere digests (ablation) | ARKVALE page-importance estimation · LongBench page-ranking simulation | recall accuracy | Helped |
| Use the mean-radius variant instead of max-radius or center-radius variants (ablation) | ARKVALE page-importance estimation · LongBench page-ranking simulation | recall accuracy | Helped |
| Use cuboid-mean digest instead of centroid digest (ablation) | ARKVALE page-importance estimation · LongBench page-ranking simulation | recall accuracy: less than 5% accuracy ... 95% accuracy for top-1 recall | Helped |
Surprising findings
- Tokens previously evicted can regain importance in later decoding steps, so permanent eviction methods risk losing vital information.
- The centroid of page keys is a poor importance estimator (under 5% top-1 recall), while bounding-volume digests reach at least 60% recall accuracy.
Limitations
- Storing a CPU backup for every KV page consumes substantial CPU memory; offloading to disk may be needed.
- Prefill-phase backup latency is hard to fully overlap with computation.
- Estimation overhead grows linearly with the number of pages, and the first two layers are excluded from eviction.
Experimental setup
Datasets: LongBench passkey retrieval
Benchmarks: HotpotQA NarrativeQA Qasper GovReport TriviaQA PassageRetrieval passkey retrieval
Metrics: F1 Rouge-L accuracy decoding latency throughput
Baselines: StreamingLLM H2O TOVA Origin (full KV cache) centroid page estimation
Builds on: vLLM StreamingLLM H2O TOVA Quest
Closest papers by meaning
- ThinK: Thinner Key Cache by Query-Driven Pruning (ICLR 2025 · similarity 0.92)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.92)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.91)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.91)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.91)
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.