Research map / Large language models / Reasoning and chain of thought

Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BalanceRoute (BR-0 prediction-free and BR-H lookahead variants), online routers that minimize data-parallel decode imbalance via a piecewise-linear F-score and two-stage admission, deployed on a 144-NPU Ascend cluster against vLLM baselines.

Problem

Load imbalance across data-parallel decode workers causes barrier idle in LLM serving, and generic routing heuristics (round-robin, JSQ, P2C, random) handle sticky, growing KV-cache workloads poorly under a sub-100 ms decision budget.

Approach

Derive a piecewise-linear F-score from the marginal effect of admissions on single-step imbalance (safe margin vs. overflow regimes), use a two-stage greedy/refined admission, and extend it with a constant-horizon lookahead H using a termination-classifier-based expected-remaining-work estimate.

Key results

• On Proprietary Data (G=8), BR-0 reduces average imbalance from 215,110 (JSQ) to 51,927 (4.1x) and raises throughput from 843 to 943 tok/s.
• Deployed BR-H with ExactMatch reaches 977 tok/s on Proprietary Data and 909 tok/s on Azure-2024, +15.4% and +14.5% over the strongest baseline.
• Throughput advantage of BR-H (oracle) over the strongest baseline grows from +13.4% at G=4 to +34.5% at G=16.

Surprising findings

• Deployed BR-H with the Survival predictor on Proprietary Data has Stage-2 MAE of 20.1 tokens versus 2.9 for ExactMatch, indicating the marginal-CDF predictor is weak on that trace.

Limitations

• Each configuration is a single replay with no replicate sweeps, so variance is unquantified.
• Deployed predictor is characterized only at G=8; behavior under deployed prediction at other cluster sizes is not measured.
• Router is a single stateful Python proxy that has not been stress-tested at substantially larger G.

Experimental setup

Datasets: Proprietary Data (8,000 requests) Azure-2024 conversation trace (filtered, 10,000 requests)

Benchmarks: Proprietary Data trace Azure-2024

Metrics: average DP imbalance TPOT P95 throughput (tok/s)

Baselines: Random Round-Robin Power-of-Two-Choices Join-Shortest-Queue

Builds on: vLLM prefill-decode disaggregation Mooncake

Closest papers by meaning

• DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving (ICLR 2026 · similarity 0.89)
• TokenRouter: Efficient Serving System for Token-Level LLM Routing (NeurIPS 2026 · similarity 0.89)
• Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.88)
• MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.87)
• Online Linear Programming for Multi-Objective Routing in LLM Serving (ICML 2026 · similarity 0.87)
• Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.86)