Research map / Large language models / Reasoning and chain of thought
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
HyGen is an interference-aware LLM serving system that co-locates online and offline requests on one instance using a linear-regression batch latency predictor, an SLO-aware profiler, and prefix-sharing-maximization offline scheduling with a fairness extension.
Problem
Dedicated clusters for online (latency-sensitive) and offline (throughput-oriented) LLM workloads waste resources during off-peak periods, but co-locating them causes interference that can violate latency SLOs.
Approach
Predict batch execution time with a linear regression model over prefill/decode token and request counts, use an SLO-aware profiler to set a batch latency budget via binary search, and fill the residual budget with offline requests scheduled in prefix-tree (DFS) order with a freshness-based fairness extension.
Key results
- HyGen improves overall serving throughput by up to 3.87x over pure online serving while meeting latency SLOs.
- HyGen achieves up to 5.84x offline throughput gain over the HyGen* hybrid baseline under the same SLO.
- The latency predictor achieves mean absolute percentage error of 1.78% (Llama2-7B) and 1.07% (Qwen-14B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Co-locate offline requests with online requests using SLO-aware scheduling | HyGen · Azure LLM Inference trace + arXiv summarization | offline throughput: up to 3.87× compared to pure online serving | Helped |
| Use HyGen's latency prediction and scheduling instead of HyGen* | HyGen · Azure LLM Inference trace + arXiv summarization | offline throughput: up to $5.84\times$ offline throughput gain | Helped |
| Co-locate online and offline workloads | HyGen · Azure LLM Inference trace + arXiv summarization | total throughput compared to pure offline serving: 84.3% | Helped |
| Add prefix sharing maximization scheduling (ablation) | HyGen · Azure LLM Inference trace + MMLU | offline throughput: up to $4\times$ offline throughput gain | Helped |
| Use HyGen with tensor parallelism and pipeline parallelism (ablation) | HyGen · Azure LLM Inference trace + arXiv summarization | offline throughput: up to $1.89\times$ | Helped |
| Replace the offline dataset with CNN/DailyMail (ablation) | HyGen · Azure LLM Inference trace + CNN/DailyMail | throughput | Helped |
| Use the Mooncake trace as the online workload (ablation) | HyGen · Mooncake trace + arXiv summarization | throughput | Helped |
| Run HyGen on an A5000 GPU with Sheared-LLaMA-2.7B (ablation) | HyGen · Azure LLM Inference trace + arXiv summarization | offline throughput: up to 2.18× | Helped |
| Run HyGen on an A5000 GPU with Sheared-LLaMA-2.7B (ablation) | HyGen · Azure LLM Inference trace + arXiv summarization | overall throughput: 1.30× | Helped |
| Use latency predictors with varying prediction accuracy (ablation) | HyGen · Azure LLM Inference trace + arXiv summarization | offline throughput under the same P99 TBT SLO | No clear effect |
| Increase online QPS (ablation) | HyGen · Online arrival-rate evaluation | offline throughput | Hurt |
Surprising findings
- Vanilla prefix-sharing-maximization scheduling can starve requests with low prefix-sharing potential, motivating the fairness extension.
Limitations
- Latency predictor assumes stable performance and may degrade under highly dynamic or adversarial inputs.
- Focuses on single-model co-location; heterogeneous models or multi-tenant settings not covered.
- Evaluation limited to specific production workloads and hardware.
Experimental setup
Datasets: Azure LLM inference trace 2023 arXiv summarization Mooncake trace CNN/DailyMail MMLU
Benchmarks: Azure LLM inference trace 2023 arXiv summarization Mooncake trace CNN/DailyMail MMLU
Metrics: TTFT (mean, P99) TBT (mean, P99) throughput (TPS, QPS) MAPE of latency predictor
Baselines: Sarathi Sarathi-offline Sarathi++ HyGen*
Builds on: vLLM Sarathi-Serve InferCept Preble
Closest papers by meaning
- OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration (ICML 2026 · similarity 0.86)
- MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.86)
- HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization (ICML 2026 · similarity 0.86)
- Efficient LLM Scheduling by Learning to Rank (NeurIPS 2024 · similarity 0.85)
- HYVE: Hybrid Views for LLM Context Engineering over Machine Data (NeurIPS 2026 · similarity 0.85)
- DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.