AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

HexGen-3: A Fully Disaggregated LLM Serving Framework with Fine-Grained Heterogeneous Resource Autoscaling

Youhe Jiang, Wenshuang Li, You Peng, Jintao Zhang, Ran Yan, Jianfei Chen, Xu Han, Fangcheng Fu, Binhang Yuan

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HexGen-3, an LLM serving framework that fully disaggregates inference phases and combines this with fine-grained, heterogeneous-resource autoscaling to cut serving cost under variable production workloads.

Problem

Serving large language models is expensive because production traffic is highly heterogeneous, so fixed resource allocations waste capacity across inference phases with different bottlenecks.

Approach

Combines disaggregated inference (separating phases such as prefill and decode) with resource autoscaling so each phase and operation can scale independently on heterogeneous hardware, based on its own bottleneck.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
disable resource allocation optimization (uniform allocation) (ablation)HexGen-3 (Homo) · static workloads, homogeneous setupper-cost throughput: -22.9% on average, up to -28.6%Hurt
also disable parallelism strategy optimization (uniform parallelism) (ablation)HexGen-3 (Homo) · static workloads, homogeneous setupper-cost throughput: further -14.6% on average, up to -25.1%Hurt
add guided generation policy to global scheduler (ablation)HexGen-3 scheduling · scheduling timescheduling time: up to 78.2% (on average 53.7%) reduction vs exhaustive searchImproved
guided scheduling vs exhaustive search (ablation)HexGen-3 scheduling · scheduling optimalitythroughput: within 3.4% differenceNo clear effect
add warm-start and cost-aware optimizations to rescheduling (ablation)HexGen-3 autoscaling · rescheduling time (resources extended 1.5x)rescheduling time: up to 65.5% (on average 63.6%) reductionImproved
add warm-start and cost-aware optimizations to rescheduling (ablation)HexGen-3 autoscaling · rescheduling optimalitythroughput: identical throughputNo clear effect
use HexGen-3 (Homo) instead of SGLangQwen3-30B-A3B serving, homogeneous H100 · static workloadsper-cost throughput: +16.6% on average, up to +21.6%Improved
use HexGen-3 (Homo) instead of MegaScale-InferQwen3-30B-A3B serving, homogeneous H100 · static workloadsper-cost throughput: +28.7% on average, up to +36.8%Improved
add attention-FFN disaggregation to heterogeneous serving (HexGen-3 vs HexGen-2)heterogeneous H100+H20 cluster · static workloadsper-cost throughput: +20.3% on average, up to +25.8%Improved
add H20 decoding-optimized GPUs (heterogeneous vs homogeneous)HexGen-3 · static workloadsper-cost throughput: +14.0% on average, up to +17.9%Improved
fine-grained attention/FFN autoscaling vs HeteroScaleHexGen-3 (Autoscale) · dynamic trace (Table 1)per-cost throughput: up to 15.8%, on average 14.8%Improved
enable autoscaling vs non-autoscaling baselinesHexGen-3 · dynamic trace (Table 1)per-cost throughput: up to 78.3%, on average 55.1%Improved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Experimental setup

Builds on: HexGen

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy