Research map / Large language models / Reasoning and chain of thought
DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DéjàVu, a pipeline-parallel LLM serving system built on a KV-cache streaming library (DéjàVuLib) that disaggregates prompt and token processing, swaps KV cache per microbatch between GPU and CPU, and replicates KV cache for fast fault recovery.
Problem
Distributed LLM serving in pipeline-parallel settings wastes GPU time due to bimodal prompt vs. token-generation latency (pipeline bubbles), overprovisioned KV cache memory, and long recovery/recomputation after failures.
Approach
Treat the KV cache as streamable state: disaggregate prompt and token machines with a resource planner, swap KV cache at microbatch granularity between GPU and CPU, and asynchronously replicate per-token KV cache to a peer worker so generation resumes from the last replicated step after a failure.
Key results
- DéjàVu sustains up to 1.88x (OPT-66B) and 2x (BLOOM-176B) higher throughput than FasterTransformer without failures.
- Microbatch swapping increases throughput by up to 1.8x by allowing 2x batch size.
- With a failure at token step 1200, DéjàVu limits latency increase to 1.24x versus 1.91x for the baseline; streaming overhead is within 2% of no-streaming for single-batch latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Disaggregate prompt processing from token generation | DéjàVu pipeline serving · LMSys, OPT-66B | throughput vs. FasterTransformer: 1.88$\times$ higher throughput | Helped |
| Disaggregate prompt processing from token generation | DéjàVu pipeline serving · LMSys, BLOOM-176B | throughput vs. FasterTransformer: 2$\times$ higher throughput | Helped |
| Swap KV cache at the microbatch level to accommodate batch size 2 · B rather than B | DéjàVu microbatch swapping · Microbatch swapping evaluation | throughput: up to 1.8$\times$ | Helped |
| Replicate KV cache and resume from the latest replicated step instead of restarting active microbatches from scratch | OPT-66B pipeline serving · Failure at token generation step 1200 | latency increase after failure: 1.91$\times$ -> 1.24$\times$ | Helped |
| Resume token generation from the latest replicated step | DéjàVu fault-tolerant serving · OPT-66B failure scenarios | runtime: 1.16$\times$ shorter runtime | Helped |
| Disaggregate prompt processing from token generation | DéjàVu · OPT-66B, LMSys trace, simulated equal machine counts | makespan: 4.2$\times$ shorter makespan than Baseline | Helped |
| Disaggregate prompt processing from token generation | DéjàVu · OPT-66B, LMSys trace, simulated equal machine counts | makespan: 2.22$\times$ shorter makespan than Baseline-DP | Helped |
| Add data parallel pipelines | Baseline pipeline serving · LMSys trace, simulated | makespan: 2.29$\times$ | Helped |
| Aggregate non-contiguous KV-cache updates in a temporary GPU buffer rather than issuing separate copies | DéjàVuLib · KV cache streaming microbenchmark | streaming performance: 95$\times$ improvement compared to baseline | Helped |
| Add the other two DéjàVuLib streaming optimizations after buffered copies | DéjàVuLib · KV cache streaming microbenchmark | streaming performance: 1.4$\times$ | Helped |
| Use larger batch sizes or longer sequence lengths with microbatch swapping | DéjàVu microbatch swapping · OPT-30B, OPT-66B, and BLOOM-176B on V100-16GB or A100-80GB | throughput | Mixed |
Surprising findings
- Prompt processing latency can be up to 106x higher than per-token generation latency, creating large pipeline bubbles.
- Larger microbatch sizes can increase pipeline bubbles for early-stopping requests, so a larger microbatch is not always beneficial.
Limitations
- Built on FasterTransformer; vLLM lacked pipeline parallelism at the time, so comparisons are against a modified FasterTransformer.
- Large-scale planner evaluation relies on a simulator due to limited budget.
- Iteration-level scheduling integration was future work.
Experimental setup
Datasets: LMSys-Chat-1M (output length distribution)
Benchmarks: LMSys trace (Poisson open-loop request arrivals)
Metrics: normalized latency (s/token) throughput makespan normalized cost cumulative latency under failure single-batch latency slowdown
Baselines: FasterTransformer vLLM SpotServe Orca
Builds on: FasterTransformer PagedAttention (vLLM) Orca iteration-level scheduling
Closest papers by meaning
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving (ICML 2026 · similarity 0.86)
- ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.86)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.85)
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.85)
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.84)
- DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.