AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Pacing Branch Parallelism in LLM Serving

Swapnil Gandhi, Siva Kumar Sastry Hari, Bill Dally, Christos Kozyrakis

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes pacing the admission of parallel decoding branches in LLM serving, instead of eager admission or fixed concurrency caps, to balance throughput and co-batched request latency.

Problem

Methods that expose intra-request parallelism let independent branches decode concurrently, but existing serving systems either admit branches eagerly, which inflates the shared decode step and degrades co-batched requests in serial stages, or use fixed caps that forgo the throughput gains.

Approach

Adaptive pacing of branch admission during serving, rather than eager execution or fixed caps, so that branch parallelism is exploited without degrading co-batched requests. (Details beyond the abstract are not available.)

Measured effects reported in the paper

What was changedWhereMeasuredEffect
admit branches only when predicted externality fits slack budget (PACE) instead of eager admissionIRP-Eager · Qwen3-32B mixed tracegoodput: 1.48x over eager; 99% SLO attainment vs below 50%Improved
eager admission of all branchesIRP-Off · high load, Qwen3-32Bgoodput: 0.88x IRP-Off goodput, 41% attainmentHurt
fixed branch caps (2 or 5) instead of adaptive admissionIRP-C2 / IRP-C5 · high loadSLO attainment: C5 45%, C2 59%Hurt
reactive MIMD global width controller instead of per-step predictionMIMD · full tracegoodput: 1.14x vs 1.77xHurt
add slack budget to PACE (ablation)PACE · Qwen3-32B tracegoodput / attainment: 0.92 -> 1.77 goodput; 48 -> 99 attainmentImproved
add per-step replanning (ablation)PACE · Qwen3-32B tracegoodput / attainment: 1.18 -> 1.77 goodput; 82 -> 99 attainmentImproved
use calibrated latency predictor instead of constant (ablation)PACE · Qwen3-32B tracegoodput: 1.12 -> 1.77; attainment 96 -> 99Improved
raise slack fraction rho from 0.8 to 0.9 or 1.0 (ablation)PACE · Qwen3-32B tracegoodput / attainment: goodput 1.77 -> 1.61 -> 1.45; attainment 99 -> 94.7 -> 91.3Hurt
lower slack fraction rho to 0.2 or below (ablation)PACE · Qwen3-32B tracegoodput: 1.77 -> 1.09 at rho 0.2Hurt
apply PACE to MoE and dense models up to 671BPACE · 5 additional modelsgoodput: 1.72-2.15x at 99% attainment (97% for Qwen2.5-72B)Improved
use SPRINT frontend with narrower branchingPACE · Qwen3-32Bgoodput: 1.45x at 98% attainmentImproved

Surprising findings

Experimental setup

Baselines: eager branch admission fixed branch concurrency caps

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy