Research map / Large language models / Reasoning and chain of thought
HexGen-3: A Fully Disaggregated LLM Serving Framework with Fine-Grained Heterogeneous Resource Autoscaling
Youhe Jiang, Wenshuang Li, You Peng, Jintao Zhang, Ran Yan, Jianfei Chen, Xu Han, Fangcheng Fu, Binhang Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HexGen-3, an LLM serving framework that fully disaggregates inference phases and combines this with fine-grained, heterogeneous-resource autoscaling to cut serving cost under variable production workloads.
Problem
Serving large language models is expensive because production traffic is highly heterogeneous, so fixed resource allocations waste capacity across inference phases with different bottlenecks.
Approach
Combines disaggregated inference (separating phases such as prefill and decode) with resource autoscaling so each phase and operation can scale independently on heterogeneous hardware, based on its own bottleneck.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| disable resource allocation optimization (uniform allocation) (ablation) | HexGen-3 (Homo) · static workloads, homogeneous setup | per-cost throughput: -22.9% on average, up to -28.6% | Hurt |
| also disable parallelism strategy optimization (uniform parallelism) (ablation) | HexGen-3 (Homo) · static workloads, homogeneous setup | per-cost throughput: further -14.6% on average, up to -25.1% | Hurt |
| add guided generation policy to global scheduler (ablation) | HexGen-3 scheduling · scheduling time | scheduling time: up to 78.2% (on average 53.7%) reduction vs exhaustive search | Improved |
| guided scheduling vs exhaustive search (ablation) | HexGen-3 scheduling · scheduling optimality | throughput: within 3.4% difference | No clear effect |
| add warm-start and cost-aware optimizations to rescheduling (ablation) | HexGen-3 autoscaling · rescheduling time (resources extended 1.5x) | rescheduling time: up to 65.5% (on average 63.6%) reduction | Improved |
| add warm-start and cost-aware optimizations to rescheduling (ablation) | HexGen-3 autoscaling · rescheduling optimality | throughput: identical throughput | No clear effect |
| use HexGen-3 (Homo) instead of SGLang | Qwen3-30B-A3B serving, homogeneous H100 · static workloads | per-cost throughput: +16.6% on average, up to +21.6% | Improved |
| use HexGen-3 (Homo) instead of MegaScale-Infer | Qwen3-30B-A3B serving, homogeneous H100 · static workloads | per-cost throughput: +28.7% on average, up to +36.8% | Improved |
| add attention-FFN disaggregation to heterogeneous serving (HexGen-3 vs HexGen-2) | heterogeneous H100+H20 cluster · static workloads | per-cost throughput: +20.3% on average, up to +25.8% | Improved |
| add H20 decoding-optimized GPUs (heterogeneous vs homogeneous) | HexGen-3 · static workloads | per-cost throughput: +14.0% on average, up to +17.9% | Improved |
| fine-grained attention/FFN autoscaling vs HeteroScale | HexGen-3 (Autoscale) · dynamic trace (Table 1) | per-cost throughput: up to 15.8%, on average 14.8% | Improved |
| enable autoscaling vs non-autoscaling baselines | HexGen-3 · dynamic trace (Table 1) | per-cost throughput: up to 78.3%, on average 55.1% | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SGLang — SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) · better on 7 of 7 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Builds on: HexGen
Closest papers by meaning
- OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration (ICML 2026 · similarity 0.90)
- MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.89)
- SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs (NeurIPS 2025 · similarity 0.89)
- Beyond Prediction: Tail-Aware Scheduling for LLM Inference (ICML 2026 · similarity 0.88)
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.88)
- Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
