Research map / Large language models / Reasoning and chain of thought

AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AugServe, a serving framework for augmented LLMs that combines state-aware value-density request scheduling with a dynamic batch-level token budget that accounts for reclaimable memory from paused requests, improving goodput over vLLM and INFERCEPT.

Problem

Augmented LLM inference with external calls under strict SLOs suffers head-of-line blocking from FCFS scheduling and from fixed batch-level token budgets that ignore reclaimable memory of paused requests, lowering effective throughput (goodput).

Approach

Ranks requests by a space-time cost value density that is updated with realized feedback at each external-call return, and sets the per-iteration token budget from free GPU memory plus reclaimable memory of paused requests under Swap/Discard policies, clipped to bounds.

Key results

• AugServe achieves a geometric mean 6.5x higher effective throughput than vLLM and 4.7x higher than INFERCEPT (abstract).
• Reduces TTFT by 95.6% vs vLLM and 96.0% vs INFERCEPT on geometric mean across settings.
• On H800 with ToolBench at 5.0 req/s, vLLM and INFERCEPT goodput drops to near zero while AugServe sustains 2.4 req/s.

Surprising findings

• Ablation shows the dynamic token budget alone raises goodput only 41.5%, while state-aware scheduling alone gives 4.2x goodput gain.

Limitations

• Prediction accuracy is modest (65% bucket accuracy on INFERCEPT, 85% on ToolBench), relying on feedback to correct errors.
• Evaluated only on a few models and GPUs with simulated/ fixed workloads of Poisson and Gamma arrivals.

Experimental setup

Datasets: INFERCEPT dataset ToolBench

Benchmarks: INFERCEPT dataset ToolBench

Metrics: goodput TTFT normalized latency SLO attainment P95 TTFT P99 TTFT

Baselines: vLLM INFERCEPT Speculative-SJF MARS

Builds on: vLLM INFERCEPT Orca MARS

Closest papers by meaning

• Fast Inference for Augmented Large Language Models (NeurIPS 2025 · similarity 0.94)
• Efficient LLM Scheduling by Learning to Rank (NeurIPS 2024 · similarity 0.89)
• Efficient Multi-round LLM Inference over Disaggregated Serving (ICML 2026 · similarity 0.88)
• TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.87)
• TimeBill: Time-Budgeted Inference for Large Language Models (AAAI 2026 · similarity 0.87)
• MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees (NeurIPS 2025 · similarity 0.87)