Research map / Large language models / Reasoning and chain of thought

Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PPD, a dynamic per-request router that lets decode nodes process multi-turn append-prefill locally using cached KV, chosen via an offline lookup table, cutting Turn 2+ TTFT in PD-disaggregated LLM serving.

Problem

Prefill-decode disaggregated LLM serving handles multi-turn conversations inefficiently: each turn re-prefills the full history on prefill nodes and re-transfers KV caches to decode nodes, causing high Turn 2+ TTFT and KV-transfer bandwidth saturation.

Approach

Append-prefill (only new tokens with cached KV) causes an order-of-magnitude smaller decode interference than full prefill, so decode nodes can safely process Turn 2+ requests locally; PPD dynamically chooses per request between this local path and the standard PD path using a score that weighs TTFT gains against TPOT degradation, from an offline-built lookup table.

Key results

• Switching from x=0 to x=1 reduces Turn 2 TTFT by 48-73% across configurations, with 1P_3D reaching 73.3% at high QPS.
• Full prefill causes ~48% TPOT slowdown at batch size 200 versus ~2% for append-prefill on Llama-3.1-8B (single H100).
• PPD keeps 100% success rate across all 27 WildChat test points, while the x=0 PD baseline fails on 13/27; PPD reports ~68% average Turn 2+ TTFT reduction on real workloads.

Surprising findings

• No single fixed routing strategy dominates TTFT, TPOT, and throughput simultaneously; 92.2% of workload-QPS combinations have different optimal configs for Turn 2 TTFT vs. TPOT.
• The x=1 advantage on Turn 2 TTFT grows with load for P-scarce configurations, while 3P_1D shows diminishing returns.

Limitations

• Lookup table is built offline and can be suboptimal when hardware or workload distributions drift from the calibration set.
• Bandwidth effects are emulated on a single 4-GPU NVLink node rather than measured on multi-node hardware.
• Comprehensive sweeps are limited to 8B models due to restart and initialization overhead.

Experimental setup

Datasets: ShareGPT WildChat synthetic multi-turn workloads (18 settings)

Benchmarks: ShareGPT WildChat synthetic multi-turn sweeps

Metrics: Turn 2 TTFT P99 TTFT TPOT throughput (tokens/s) success rate average query latency

Baselines: PD disaggregation (x=0) Full AP-to-D (x=1) Replica AMPD

Builds on: prefill-decode disaggregation vLLM disaggregated serving chunked prefill TetriInfer

Closest papers by meaning

• Efficient Multi-round LLM Inference over Disaggregated Serving (ICML 2026 · similarity 0.89)
• TokenRouter: Efficient Serving System for Token-Level LLM Routing (NeurIPS 2026 · similarity 0.86)
• RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse (ICML 2026 · similarity 0.85)
• When RL Meets Adaptive Speculative Training: A Unified Training-Serving System (ICML 2026 · similarity 0.85)
• MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees (NeurIPS 2025 · similarity 0.85)
• Evaluating Test-Time Scaling of General LLM Agents (NeurIPS 2026 · similarity 0.85)