Research map / Architectures and efficiency / RNNs and normalization
WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
Chiheng Lou, Sheng Qi, Rui Kang, Yong Zhang, Chen Sun, pengcheng wang, Xuanzhe Liu, Xin Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add one-for-many GPU prewarming (weights in GPU memory) versus on-demand engine creation | WarmServe · Llama2-7B/13B/70B cold-start | TTFT: 60.13x-87.63x TTFT reduction | Improved |
| use WarmServe instead of SLLM-GPU autoscaling | multi-LLM serving · AzureConv-based workload, RPS 10-25 | P99 TTFT: 1.53x-50.79x reduction | Improved |
| add proactive prewarming into unused KV cache of GPUs about to be released (ablation) | WarmServe · AzureConv-based workload | tail TTFT: 1.03x-32.87x reduction | Improved |
| use WarmServe instead of MuxServe GPU sharing under heavy load | multi-LLM serving · RPS 15, alpha=0.5 | P95/P99 TTFT: MuxServe 34.5x and 134.0x higher P95 and P99 TTFT | Improved |
| add model prewarming (ablation) | WarmServe · RPS 25, 100ms TTFT threshold | fraction of requests within 100ms: without it 0.15x of baseline | Improved |
| add model placement algorithm instead of round robin (ablation) | WarmServe · RPS 25, 100ms TTFT threshold | fraction of requests within 100ms: without it 0.29x of baseline | Improved |
| use 5-minute prediction window instead of 3 or 40 minutes (ablation) | WarmServe · RPS 25, 100ms TTFT threshold | fraction of requests within 100ms: 3 min 0.46x, 40 min 0.30x of 5-min baseline | Improved |
| add one-for-many prewarming versus predictive autoscaling without prewarming at high load | 512-GPU simulation · RPS 800 | P99 TTFT (s): 4.42 -> 0.38 | Improved |
| use predictive autoscaling instead of prewarming at light load | 512-GPU simulation · RPS 320 and 480 | P99 TTFT (s): 0.18 vs 0.18 and 0.25 vs 0.25 | No clear effect |
Closest papers by meaning
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.93)
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving (ICML 2026 · similarity 0.90)
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.90)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.90)
- QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration (ICML 2025 · similarity 0.89)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
