Research map / Architectures and efficiency / RNNs and normalization
Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching
Yanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng, Chao Wang, Jiesheng Wu, Feng Lyu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prefetch next KV blocks into L2 cache during compute | vLLM XFormers attention kernel · Llama2-7B, batch 64, 4K output tokens, H20 | L2 cache hit rate (%): 0.06 -> 43.70 | Improved |
| prefetch next KV blocks into L2 cache during compute | vLLM XFormers attention kernel · Llama2-7B, batch 64, 4K output tokens, H20 | Stall Long Scoreboard (cycles): 21.34 -> 4.13 | Improved |
| prefetch next KV blocks into L2 cache during compute | vLLM XFormers attention kernel · 4 models, batch 64, 4K output tokens, H20 | kernel speedup: 1.84× to 2.15× | Improved |
| prefetch next KV blocks into L2 cache during compute | vLLM XFormers backend end-to-end · single H20, 2048 output tokens, batch 16-128 | throughput vs XFormers: up to 51%, 57%, 41%, 41% (Llama2-7B, Llama3-8B, Qwen2.5-7B, Qwen2.5-14B) | Improved |
| use prefetching XFormers instead of FlashAttention-3 backend | vLLM end-to-end · single H20, Qwen2.5-7B (GQA 7:1) | throughput vs FA3: 2-5% regression | Hurt |
| use prefetching XFormers instead of FlashAttention-3 backend | vLLM end-to-end · single H20, Llama2-7B, Llama3-8B, Qwen2.5-14B | throughput vs FA3: max gains 110%, 15%, 7% | Improved |
| use GQA with a higher Q:KV head ratio (ablation) | prefetching method · single H20 | relative gain of prefetching | Hurt |
| increase output sequence length (ablation) | prefetching method · Llama3-8B, Qwen2.5-7B, single H20 | speedup over XFormers | Improved |
| increase batch size to 128 with 8K output tokens (ablation) | prefetching method · single H20 | speedup over XFormers | Hurt |
| scale tensor parallelism from 2 to 8 GPUs (ablation) | prefetching method · Llama3-8B, Qwen2.5-14B, batch 64, 4096 output tokens | throughput gain over XFormers: 4%-59% (Llama3-8B), 4%-76% (Qwen2.5-14B) | Hurt |
| use FlashAttention-3 instead of XFormers with 4/8-GPU TP | Llama3-8B · multi-GPU TP, batch 64 | throughput: 3-7% regression | Hurt |
Closest papers by meaning
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.91)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.91)
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · similarity 0.91)
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.90)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.90)
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
