Research map / Large language models / Reasoning and chain of thought

Efficient Multi-round LLM Inference over Disaggregated Serving

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AMPD, a disaggregated serving framework for multi-round LLM workflows that adaptively routes incremental prefill tasks (local vs remote) and reorders prefill queues under TTFT SLOs, with an ILP-based deployment planner.

Problem

Serving multi-round LLM workflows (agents, iterative RAG) under prefill-decode disaggregation is hard because incremental prefills interleave with decoding, and existing systems lack adaptive routing/scheduling and workload-aware deployment planning.

Approach

Adaptive SLO-aware routing decides per incremental prefill whether to run it locally on the bound decode worker or remotely on a prefill worker, combined with a TTFT-aware lookahead reordering of the prefill queue and an ILP planner that picks replica counts and TP degrees using simulated P95 latency.

Key results

• AMPD improves SLO attainment by up to 967.54% over Dynamo and up to 3435.1% over vLLM (67.29% and 339.74% on average).
• Adaptive routing routes 13.9%-31.7% of prefill tasks locally and raises SLO attainment by 27.37%-350%; prefill reordering adds 13.42%-14.81%.
• Planner overhead is about one minute over 256 GPUs, and its top-ranked configuration matched real-system best in all reported experiments.

Surprising findings

• Co-located vLLM can beat disaggregated Dynamo on GAIA because it gives competitive TTFT without a pronounced ITL penalty.

Limitations

• ILP minimizes P95 latency as a surrogate because SLO attainment is binary and not solvable directly by LP solvers.
• Planner uses a fixed workload and hardware condition, so deployments can become suboptimal under workload drift or elastic changes.
• Evaluation covers only four traces and three models on H20 GPUs.

Experimental setup

Datasets: ToolBench GAIA HotpotQA DuReader

Benchmarks: ToolBench trace GAIA trace HotpotQA trace DuReader trace

Metrics: SLO attainment TTFT ITL end-to-end latency

Baselines: Dynamo vLLM vLLM-Continuum

Builds on: NVIDIA Dynamo vLLM ReAct InferCept

Closest papers by meaning

• LLM Query Scheduling with Prefix Reuse and Latency Constraints (NeurIPS 2025 · similarity 0.89)
• Fast Inference for Augmented Large Language Models (NeurIPS 2025 · similarity 0.89)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.89)
• Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling (NeurIPS 2025 · similarity 0.89)
• Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving (ICML 2026 · similarity 0.89)
• Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.88)