Research map / Multi-agent systems and games / LLM agents
KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KVFlow, a workflow-aware KV cache manager for LLM agent workflows that evicts prefix KV nodes by steps-to-execution from an Agent Step Graph and prefetches upcoming agents' KV from CPU, reporting speedups over SGLang with HiCache.
Problem
LRU eviction of prefix KV caches in LLM serving systems discards caches of agents that are about to be reused in multi-agent agentic workflows, causing cache misses and recomputation or CPU-GPU swapping overhead.
Approach
Abstract the agent execution schedule as an Agent Step Graph that yields a steps-to-execution value per agent; use it to assign priorities to radix-tree KV nodes for eviction, and proactively prefetch KV from CPU for agents scheduled in the next step, with status-aware scheduling to avoid stalls.
Key results
- Under 8192/32/32 on A10G with Llama-3.1-8B, KVFlow outperforms SGLang w/ HiCache by 1.83x and GPU-only SGLang by 2.91x (single workflow).
- In high-concurrency settings on H100, KVFlow achieves up to 1.25x speedup over SGLang and HiCache, and up to 2.19x over naive LRU-based HiCache.
- On realistic PEER-based workflows, KVFlow achieves up to 1.12x and 1.08x speedups over SGLang and HiCache respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use workflow-aware eviction and proactive KV prefetching instead of SGLang w/ HiCache | SGLang serving 10-agent sequential workflows · A10G, 8192/32/32 | speedup: 1.83× | Helped |
| use workflow-aware eviction and proactive KV prefetching instead of GPU-only SGLang | SGLang serving 10-agent sequential workflows · A10G, 8192/32/32 | speedup: 2.91× | Helped |
| increase fixed prompt length from 4096 to 8192 tokens | KVFlow · 10-agent sequential workflow | average speedup over SGLang: 1.28× to 1.48× | Helped |
| increase output token count | KVFlow · 10-agent sequential workflow | relative speedup | Hurt |
| use SGLang HiCache with CPU backup and reactive loading | SGLang · High concurrency, 1024 fixed prompt tokens, 64 workflows | relative performance vs. SGLang: 0.57× | Hurt |
| use workflow-aware eviction and proactive prefetching instead of reactive HiCache loading | SGLang HiCache · High-concurrency workflows | performance gain: up to 2.19× | Helped |
| use KVFlow instead of SGLang | SGLang serving PEER Financial QA workflows · PEER Financial QA | speedup: up to 1.12× | Helped |
| use KVFlow instead of SGLang w/ HiCache | SGLang serving PEER Financial QA workflows · PEER Financial QA | speedup: up to 1.08× | Helped |
Surprising findings
- SGLang w/ HiCache performs worse than GPU-only SGLang under high concurrency, reaching only 0.57x of SGLang at 1024 fixed tokens with 64 concurrent workflows.
- On H100 with large contexts, HiCache shows marginal or degraded performance relative to GPU-only SGLang.
Limitations
- Evaluation uses synthetic prompts for the main single-workflow experiments and only a moderate-scale realistic workload.
- Gains diminish as output token count grows because decoding latency dominates.
- Fragmented KV storage in SGLang limits PCIe bandwidth utilization, which KVFlow does not resolve.
Experimental setup
Datasets: PEER Financial QA dataset (workflow input) synthetic random token prompts
Benchmarks: synthetic 10-agent sequential workflow PEER-style multi-agent workflows
Metrics: end-to-end latency speedup
Baselines: SGLang SGLang w/ HiCache
Builds on: SGLang RadixAttention hierarchical radix cache (HiCache) PEER
Closest papers by meaning
- KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems (NeurIPS 2025 · similarity 0.91)
- Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows (NeurIPS 2026 · similarity 0.90)
- LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents (ICML 2026 · similarity 0.90)
- Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents (NeurIPS 2025 · similarity 0.89)
- CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control (ICML 2026 · similarity 0.89)
- RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.