Research map / Architectures and efficiency / RNNs and normalization
Pacing Branch Parallelism in LLM Serving
Swapnil Gandhi, Siva Kumar Sastry Hari, Bill Dally, Christos Kozyrakis
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pacing the admission of parallel decoding branches in LLM serving, instead of eager admission or fixed concurrency caps, to balance throughput and co-batched request latency.
Problem
Methods that expose intra-request parallelism let independent branches decode concurrently, but existing serving systems either admit branches eagerly, which inflates the shared decode step and degrades co-batched requests in serial stages, or use fixed caps that forgo the throughput gains.
Approach
Adaptive pacing of branch admission during serving, rather than eager execution or fixed caps, so that branch parallelism is exploited without degrading co-batched requests. (Details beyond the abstract are not available.)
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| admit branches only when predicted externality fits slack budget (PACE) instead of eager admission | IRP-Eager · Qwen3-32B mixed trace | goodput: 1.48x over eager; 99% SLO attainment vs below 50% | Improved |
| eager admission of all branches | IRP-Off · high load, Qwen3-32B | goodput: 0.88x IRP-Off goodput, 41% attainment | Hurt |
| fixed branch caps (2 or 5) instead of adaptive admission | IRP-C2 / IRP-C5 · high load | SLO attainment: C5 45%, C2 59% | Hurt |
| reactive MIMD global width controller instead of per-step prediction | MIMD · full trace | goodput: 1.14x vs 1.77x | Hurt |
| add slack budget to PACE (ablation) | PACE · Qwen3-32B trace | goodput / attainment: 0.92 -> 1.77 goodput; 48 -> 99 attainment | Improved |
| add per-step replanning (ablation) | PACE · Qwen3-32B trace | goodput / attainment: 1.18 -> 1.77 goodput; 82 -> 99 attainment | Improved |
| use calibrated latency predictor instead of constant (ablation) | PACE · Qwen3-32B trace | goodput: 1.12 -> 1.77; attainment 96 -> 99 | Improved |
| raise slack fraction rho from 0.8 to 0.9 or 1.0 (ablation) | PACE · Qwen3-32B trace | goodput / attainment: goodput 1.77 -> 1.61 -> 1.45; attainment 99 -> 94.7 -> 91.3 | Hurt |
| lower slack fraction rho to 0.2 or below (ablation) | PACE · Qwen3-32B trace | goodput: 1.77 -> 1.09 at rho 0.2 | Hurt |
| apply PACE to MoE and dense models up to 671B | PACE · 5 additional models | goodput: 1.72-2.15x at 99% attainment (97% for Qwen2.5-72B) | Improved |
| use SPRINT frontend with narrower branching | PACE · Qwen3-32B | goodput: 1.45x at 98% attainment | Improved |
Surprising findings
- Eager admission of parallel branches can degrade co-batched requests in serial stages, and conservative fixed caps forgo the throughput that motivated exposing branches.
Experimental setup
Baselines: eager branch admission fixed branch concurrency caps
Closest papers by meaning
- Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale (NeurIPS 2026 · similarity 0.88)
- TokenRouter: Efficient Serving System for Token-Level LLM Routing (NeurIPS 2026 · similarity 0.85)
- DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving (ICLR 2026 · similarity 0.85)
- MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.85)
- HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization (ICML 2026 · similarity 0.84)
- Beyond Prediction: Tail-Aware Scheduling for LLM Inference (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
