Research map / Large language models / Reasoning and chain of thought

MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MAPS, a memory-aware predictive scheduler for PD-disaggregated LLM serving that uses a device-side LoRA-tuned speculative length predictor overlapped with cloud prefill, conformal calibration for upper bounds, and hierarchical global-local scheduling.

Problem

Decode instances in prefill-decode disaggregated LLM serving suffer persistent load imbalance and head-of-line blocking because output lengths are unknown at dispatch time, and existing routing (round-robin, least-request) ignores decode demand.

Approach

Combine a device-hosted speculative output-length predictor (overlapped with cloud prefill) with one-sided conformal calibration to get coverage-guaranteed upper bounds, then use these bounds for memory-feasible, minimum-queue decoder routing plus local shortest-job-first reordering.

Key results

• Reduces average end-to-end latency by 42.6% and tail (P99) E2E latency by up to 84.8% versus three baselines across workloads and models.
• SP-7B reduces MAE by 11.1% and 21.1% over PO-IT and S3 on Alpaca (MAE 56 vs 63 and 71).
• SP+UAC lowers underestimation probability from 64% to 9% on the evaluated setting.

Surprising findings

• Routing to the decoder with maximum remaining KV capacity (MC) significantly degrades latency versus minimum-queue routing, with P99 up to 1.11x higher on Qwen2.5-3B.
• Even the lightweight SP-160M predictor, with much lower accuracy, still outperforms all baselines after calibration, and MAPS achieves slightly lower mean latency than an oracle scheduler.

Limitations

• Evaluated on a small deployment (one prefiller, two decoders on six A6000 GPUs); larger scale is untested.
• Chain-of-thought workloads with highly variable intermediate tokens are not addressed by the predictor.
• Predictor is fine-tuned only on Alpaca, so accuracy may degrade on code generation or agentic tool-use workloads.

Experimental setup

Datasets: Alpaca ShareGPT BurstGPT

Benchmarks: ShareGPT BurstGPT Alpaca

Metrics: E2E latency (mean, P99) TTFT TPOT max ITL MAE Acc-50 Acc-100 underestimation probability time-weighted queue length

Baselines: vLLM SGLang Llumnix PO-IT S3

Builds on: PagedAttention DistServe Llumnix PO-IT S3 conformal prediction LoRA

Closest papers by meaning

• Beyond Prediction: Tail-Aware Scheduling for LLM Inference (ICML 2026 · similarity 0.90)
• OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration (ICML 2026 · similarity 0.90)
• DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving (ICLR 2026 · similarity 0.89)
• Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice (NeurIPS 2026 · similarity 0.88)
• DON’T STOP ME NOW: EMBEDDING BASED SCHEDULING FOR LLMS (ICLR 2025 · similarity 0.87)
• Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale (NeurIPS 2026 · similarity 0.87)