Research map / Large language models / Reasoning and chain of thought

OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OServe, an LLM serving system that jointly searches heterogeneous model deployments and request-to-replica assignment via a two-level flow-network scheduler, and switches deployments on predicted workload changes using LSTM forecasting and greedy GPU-to-GPU parameter transfer.

Problem

LLM serving workloads are spatially heterogeneous (requests with different compute/memory demands) and temporally shifting, but existing systems use homogeneous, static model deployments, leading to suboptimal latency and throughput.

Approach

Combine a max-flow based workload assignment (lower level) with flow-guided iterative GPU reallocation and parallelism search (upper level) to produce heterogeneous replicas, and predict per-type arrival rates with an LSTM so the deployment can be switched via fast GPU-to-GPU parameter transfer rather than reloading from disk.</1>

Key results

• Improves P99 latency and throughput by up to 2.0x (average 1.5x) over vLLM (static) and up to 1.5x (average 1.3x) over vLLM (reload).
• Outperforms Llumnix by 1.32-1.51x in P99 latency and throughput on Llama-30B and Llama2-70B, and improves over Dynamo+vLLM by 12-20%.
• LSTM workload predictor achieves an average RRMSE of 5.045% with under 30 ms prediction time; switching cost reduced to around 10 seconds versus over 50 seconds for reloading.

Surprising findings

• Heuristic scheduling is within 6% P99 latency of exhaustive search while running in about 12 s versus about 50 s on 16 GPUs.

Limitations

• Heuristic greedy switch plan and search are not guaranteed optimal.
• Extension to heterogeneous GPU types is left for future work.
• Evaluated only on H100 clusters with OPT and Llama models up to 70B.

Experimental setup

Datasets: Azure Public Dataset BurstGPT ShareGPT WildChat MMLU

Benchmarks: Azure Public Dataset traces (Trace 1 and Trace 2)

Metrics: P99 latency average latency P90 latency throughput TTFT TBT RRMSE

Baselines: vLLM (static) vLLM (reload) Llumnix Dynamo+vLLM

Builds on: vLLM Vidur HexGen Llumnix

Closest papers by meaning

• MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.90)
• Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.89)
• SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.88)
• Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice (NeurIPS 2026 · similarity 0.88)
• HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization (ICML 2026 · similarity 0.87)
• DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving (ICLR 2026 · similarity 0.87)