Research map / Large language models / Reasoning and chain of thought

TokenRouter: Efficient Serving System for Token-Level LLM Routing

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TokenRouter, a serving system for token-level LLM routing that uses a request-centric route-send-receive API, asynchronous model-centric subservers, and a delayed-batching scheduler with a DTMC-derived throughput-optimal threshold.

Problem

Existing LLM serving systems assume single-model execution and handle token-level routing (where the target model switches at every decoding step) inefficiently, suffering from step desynchronization, batch admission delay, and high implementation complexity.

Approach

Model-centric asynchronous subservers (one per LLM) with a tri-loop execution structure, a pending-state handoff-resume mechanism that avoids re-running prefix matching, and a delayed-batching scheduler whose threshold is chosen by a Markov-chain throughput model; developers only implement route, send, and receive functions.

Key results

• Across 15 algorithm-workload combinations, TokenRouter improves throughput by 2.01-64.15x over the stronger baseline (2.01-64.15x; 2.76x overall at concurrency 8 for R2R).
• At concurrency 16, TokenRouter achieves 18.58x higher throughput than official R2R at concurrency 1 while keeping 1.13x higher per-user speed.
• Under original algorithm settings at concurrency 4, TokenRouter reduces latency 2.78-21.61x and improves throughput 2.73-21.97x over official implementations.

Surprising findings

• Std. Serving (SGLang-based token-level routing) spends most per-step time on prefix matching and RadixCache maintenance rather than model inference (SLM inference only 4.20% of step latency).

Limitations

• Throughput model assumes geometric distribution of tokens between consecutive sends; corner cases violating this are left for future work.
• Evaluation is on a single 8xA100 server with specific model pairs and only a limited set of routing algorithms (R-Stitch official code not released).

Experimental setup

Datasets: AIME2024 SWE-Smith AMC23 CommonsenseQA GSM8K AIME

Benchmarks: AIME2024 SWE-Smith AMC23 CommonsenseQA GSM8K

Metrics: throughput (token/s) end-to-end latency TTFT per-user speed accuracy

Baselines: Official Code (R2R, CITER, Co-LLM) Std. Serving (SGLang-based) LLM-only vLLM RouteLLM

Builds on: SGLang R2R R-Stitch CITER Co-LLM

Closest papers by meaning

• Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.89)
• MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees (NeurIPS 2025 · similarity 0.89)
• Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale (NeurIPS 2026 · similarity 0.89)
• R2-Router: A New Paradigm for LLM Routing with Reasoning (ICML 2026 · similarity 0.89)
• Efficient LLM Scheduling by Learning to Rank (NeurIPS 2024 · similarity 0.88)
• Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving (NeurIPS 2025 · similarity 0.88)