Research map / Large language models / Reasoning and chain of thought
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks LLM serving across heterogeneous cloud GPU types and formulates a MILP scheduler that jointly chooses GPU composition, deployment configuration (DP/TP/PP), and workload assignment under budget and GPU availability constraints.
Problem
Serving diverse LLM requests on homogeneous GPUs is cost-inefficient, and existing heterogeneous serving systems ignore real-time cloud GPU availability and user price budgets as well as workload heterogeneity.
Approach
Co-optimize which heterogeneous GPUs to rent, how to parallelize them into model replicas, and how to split workload types across replicas via a MILP minimizing makespan under budget and availability constraints, with binary search and knapsack-based feasibility checks for scalability.
Key results
- Method improves system throughput by up to 41% (25% on average) and reduces percentile latency by up to 54% (20% on average) versus homogeneous and heterogeneous baselines.
- Versus HexGen with uniform composition, HexGen loses up to 35% (29% average); even with optimal composition the method outperforms HexGen by up to 18% (14% average).
- Against Helix on a $15/h budget, the method outperforms Helix by 25-35%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly optimize GPU composition, deployment configuration, and workload assignment | LLM serving with heterogeneous cloud GPUs · Across evaluated traces, GPU availabilities, budgets, and models | throughput: up to 41% (25% on average) | Helped |
| jointly optimize GPU composition, deployment configuration, and workload assignment | LLM serving with heterogeneous cloud GPUs · Across evaluated traces, GPU availabilities, budgets, and models | percentile latency: up to 54% (20% on average) | Helped |
| use workload-aware scheduling with the optimal GPU composition | HexGen · Heterogeneous serving comparisons | performance: up to 18% and on average 14% | Helped |
| optimize GPU composition instead of renting GPUs uniformly across six types (ablation) | Ours scheduling algorithm · Llama3-70B, traces 1 and 2 | throughput: up to 27% (average: 20%) | Helped |
| optimize deployment configuration per model replica instead of applying TP uniformly (ablation) | Ours scheduling algorithm · Llama3-70B, traces 1 and 2 | throughput: up to 34% (average: 33%) | Helped |
| use heterogeneous-aware workload assignment instead of Round-Robin assignment (ablation) | Ours scheduling algorithm · Llama3-70B, traces 1 and 2 | throughput: up to 32% (average: 29%) | Helped |
| use binary search with feasibility checks and knapsack approximation instead of exhaustive MILP search | Scheduling algorithm · Large-scale search | search time: approximately a 4× reduction | Helped |
| use binary search instead of exhaustive MILP search | Scheduling algorithm · Large-scale search | algorithm performance: deviations of less than 1% | No clear effect |
| jointly optimize resource allocation for multiple models | Ours scheduling algorithm · Multi-model serving: Llama3-8B and Llama3-70B | performance: up to a 35% (average: 23%) | Helped |
| replan GPU composition after a workload distribution shift | Heterogeneous serving on Azure-Trace · Azure-Trace workload surge | throughput: 23.70 req/s -> 29.61 req/s (25%↑) | Helped |
| replan after a GPU availability drop | Heterogeneous serving on Azure-Trace · 4 H100s down | throughput: 20.80 req/s -> 22.85 req/s (10%↑) | Helped |
Surprising findings
- Workstation GPUs (A40, A6000, L40) deliver ~1.2x memory bandwidth and ~1.8x memory capacity per unit price over data-center GPUs for memory-intensive workloads.
- For Llama3-8B, consumer 4090 GPUs give roughly 1.9x higher memory bandwidth per unit price than A100/H100.
Limitations
- Throughput estimates come from one-time profiling with 4-7% estimation error.
- MILP worst-case complexity grows exponentially with the number of configurations.
- Homogeneous baselines assume unlimited GPUs, and the gap narrows as budgets rise due to cloud availability limits.
Experimental setup
Datasets: Swiss AI Center traces ShareGPT WildChat Azure-Trace
Benchmarks: Swiss AI Center traces Azure-Trace WildChat
Metrics: throughput percentile latency (p10-p100) cost efficiency (throughput per unit price)
Baselines: HexGen Helix Homogeneous H100 Homogeneous A6000 Homogeneous 4090
Builds on: HexGen Helix Vidur vLLM
Closest papers by meaning
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving (ICML 2026 · similarity 0.91)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.90)
- QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration (ICML 2025 · similarity 0.90)
- OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration (ICML 2026 · similarity 0.89)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.89)
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.