Research map / Architectures and efficiency / RNNs and normalization
DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Mark D. Hill, Murali Annavaram
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DuetServe adaptive SM multiplexing instead of vLLM (short output, OSL=64) | vLLM · Synthetic ISL=4096 OSL=64 | throughput (req/s): 10.0 -> 12.8 | Improved |
| use DuetServe instead of vLLM (OSL=1024) | vLLM · Synthetic ISL=4096 OSL=1024 | throughput (req/s): 8.8 -> 9.8 | Improved |
| use DuetServe instead of vLLM (decode-heavy, OSL=2048) | vLLM · Synthetic ISL=4096 OSL=2048 | throughput (req/s): 6.9 -> 7.2 | No clear effect |
| use DuetServe instead of SGLang-Default | SGLang-Default · Azure-Code QPS=16 | output request throughput (req/s): 12.43 -> 13.57 | Improved |
| use DuetServe instead of vLLM at QPS=5 | vLLM · Mooncake Conversation | output request throughput (req/s): 2.14 -> 2.72 | Improved |
| use adaptive SM partitioning instead of PD disaggregation (Dynamo, 8 GPUs) | Dynamo · Azure-Conv Qwen3-32B QPS=24 | throughput (req/s): 5.69 -> 8.02 | Improved |
| use adaptive SM partitioning instead of PD disaggregation (Dynamo, 8 GPUs) | Dynamo · Azure-Conv Qwen3-32B QPS=24 | TTFT (s): 110.2 -> 58.9 | Improved |
| use adaptive SM partitioning instead of PD disaggregation (Dynamo, 8 GPUs) | Dynamo · Azure-Conv Qwen3-32B QPS=24 | TBT (ms): 23.1 -> 104.7 | Hurt |
| use adaptive SM partitioning instead of PD disaggregation (Dynamo, 8 GPUs) | Dynamo · Azure-Conv Qwen3-32B QPS=24 | GPU utilization (%): 74.6% -> 93.5% | Improved |
| use adaptive scheduling instead of static SM partitions (Sd22-Sp44, Sd33-Sp33, Sd44-Sp22) (ablation) | DuetServe · Azure-Code/Azure-Conv/Mooncake, Qwen3-8B/14B | throughput | Improved |
| use a more conservative (overestimating) roofline decode latency model (ablation) | DuetServe · Roofline model, small TPC decode | partition quality | No clear effect |
| add multi-GPU TP2 DuetServe vs Dynamo-1P1D | Dynamo-1P1D · Azure-Code TP=2 | throughput | Improved |
Closest papers by meaning
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving (ICML 2026 · similarity 0.91)
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.88)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.87)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.87)
- ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive (NeurIPS 2025 · similarity 0.87)
- EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
