Research map / Large language models / Reasoning and chain of thought
HexGen: Generative Inference of Large Language Model over Heterogeneous Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HexGen, a distributed LLM inference engine with asymmetric tensor and pipeline parallelism and a genetic-algorithm scheduler that serves Llama-2 (70B) across heterogeneous GPUs and cross-datacenter networks.
Problem
Serving LLM generative inference cheaply over heterogeneous GPUs connected by heterogeneous (including cross-region) networks, where existing systems assume symmetric homogeneous parallel layouts.
Approach
Allow each pipeline stage to have its own number of layers and tensor-parallel degree, and find the device-to-pipeline assignment with a two-phase search: a genetic algorithm over GPU partitions with mutations (merge, split, swap) and a dynamic program for stage layout per pipeline, scored by an analytic cost model and the AlpaServe simulator.
Key results
- Under the same budget (~$65/hour), HexGen on heterogeneous GPUs achieves up to 2.3x lower latency deadlines and tolerates up to 4x higher request rates than the FlashAttention homogeneous A100 baseline.
- With half the budget, HexGen maintains performance similar to the homogeneous baseline.
- HexGen achieves up to 3.5x lower latency deadlines and up to 10x higher request rates than Petals under the half-price budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Deploy inference across heterogeneous GPUs with asymmetric parallelism and scheduling instead of a homogeneous centralized deployment | Llama-2 (70B) inference · Heterogeneous full-price vs homogeneous deployment | latency deadline: up to 2.3× lower | Helped |
| Deploy inference across heterogeneous GPUs with asymmetric parallelism and scheduling instead of a homogeneous centralized deployment | Llama-2 (70B) inference · Heterogeneous full-price vs homogeneous deployment | peak request rate: up to 4× higher ($2× higher on average) | Helped |
| Use a heterogeneous deployment at half the budget | HexGen · Heterogeneous half-price vs homogeneous deployment | inference service performance | No clear effect |
| Add asymmetric parallel groups | HexGen · Heterogeneous full-price deployment | latency deadline: up to 1.8× improvement | Helped |
| Add asymmetric parallel groups | HexGen · Heterogeneous full-price deployment | peak request rate: up to 2× higher ($1.5× on average) | Helped |
| Use static scheduling with asymmetric parallelism instead of Petals' decentralized inference approach | Llama-2 (70B) inference · Heterogeneous half-price deployment | latency deadline: up to 3.5× lower | Helped |
| Use static scheduling with asymmetric parallelism instead of Petals' decentralized inference approach | Llama-2 (70B) inference · Heterogeneous half-price deployment | request rate: up to 10× higher | Helped |
| Use heterogeneous full-price deployment instead of Huggingface-TGI's homogeneous deployment | Llama-2 (70B) inference · End-to-end system comparison | latency deadline: up to 1.25× lower | Helped |
| Use heterogeneous full-price deployment instead of Huggingface-TGI's homogeneous deployment | Llama-2 (70B) inference · End-to-end system comparison | request rate | No clear effect |
| Use constrained mutation rather than random mutation in the scheduler | HexGen scheduler · Heterogeneous full-price and half-price scenarios | SLO attainment: around 26% more | Helped |
| Remove the constrained mutation policy and use random mutation (ablation) | HexGen scheduler · Heterogeneous full-price and half-price scenarios | convergence | Hurt |
| Use fully asymmetric allocation rather than symmetric parallel execution | Llama-2 (70B) inference · Heterogeneous 8-GPU case study | speedup: 2× and 19× speedup | Helped |
Surprising findings
- Asymmetric parallel layouts achieved up to 19x speedup over symmetric layouts in a small 3-instance case study where pure TP or PP ran out of memory.
- Pipeline parallelism across slow inter-machine links combined with intra-machine tensor parallelism was preferred by the scheduler, with cross-region links avoided.
Limitations
- No advanced batching policy integrated, which limits end-to-end performance versus HuggingFace-TGI.
- Scheduler relies on an analytic cost model and the scheduling problem is NP-hard, solved heuristically.
- Cost model simplifies computation to memory scan and FLOPs and ignores other dynamic factors.
Experimental setup
Datasets: real-world prompts (unnamed)
Benchmarks: SLO attainment under Poisson request workloads on Llama-2 (70B)
Metrics: SLO attainment minimum latency deadline at 99% attainment peak request rate search convergence
Baselines: FlashAttention (homogeneous) HuggingFace-TGI Petals HexGen without asymmetric parallelism random mutation search
Builds on: AlpaServe FlashAttention Petals PagedAttention
Closest papers by meaning
- Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference (ICLR 2025 · similarity 0.86)
- SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models (ICML 2025 · similarity 0.86)
- Attention-Level Speculation (ICML 2025 · similarity 0.86)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.86)
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.85)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.