AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching

Yanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng, Chao Wang, Jiesheng Wu, Feng Lyu

AAAI 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
prefetch next KV blocks into L2 cache during computevLLM XFormers attention kernel · Llama2-7B, batch 64, 4K output tokens, H20L2 cache hit rate (%): 0.06 -> 43.70Improved
prefetch next KV blocks into L2 cache during computevLLM XFormers attention kernel · Llama2-7B, batch 64, 4K output tokens, H20Stall Long Scoreboard (cycles): 21.34 -> 4.13Improved
prefetch next KV blocks into L2 cache during computevLLM XFormers attention kernel · 4 models, batch 64, 4K output tokens, H20kernel speedup: 1.84× to 2.15×Improved
prefetch next KV blocks into L2 cache during computevLLM XFormers backend end-to-end · single H20, 2048 output tokens, batch 16-128throughput vs XFormers: up to 51%, 57%, 41%, 41% (Llama2-7B, Llama3-8B, Qwen2.5-7B, Qwen2.5-14B)Improved
use prefetching XFormers instead of FlashAttention-3 backendvLLM end-to-end · single H20, Qwen2.5-7B (GQA 7:1)throughput vs FA3: 2-5% regressionHurt
use prefetching XFormers instead of FlashAttention-3 backendvLLM end-to-end · single H20, Llama2-7B, Llama3-8B, Qwen2.5-14Bthroughput vs FA3: max gains 110%, 15%, 7%Improved
use GQA with a higher Q:KV head ratio (ablation)prefetching method · single H20relative gain of prefetchingHurt
increase output sequence length (ablation)prefetching method · Llama3-8B, Qwen2.5-7B, single H20speedup over XFormersImproved
increase batch size to 128 with 8K output tokens (ablation)prefetching method · single H20speedup over XFormersHurt
scale tensor parallelism from 2 to 8 GPUs (ablation)prefetching method · Llama3-8B, Qwen2.5-14B, batch 64, 4096 output tokensthroughput gain over XFormers: 4%-59% (Llama3-8B), 4%-76% (Qwen2.5-14B)Hurt
use FlashAttention-3 instead of XFormers with 4/8-GPU TPLlama3-8B · multi-GPU TP, batch 64throughput: 3-7% regressionHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy