Research map / Large language models / Reasoning and chain of thought

Routing, Cascades, and User Choice for LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Models a Stackelberg game between an LLM provider routing tasks to standard vs reasoning models (with cascades) and a user who re-prompts or churns, deriving optimal routing, misalignment gaps, and the incentive to throttle latency.

Problem

LLM routing and cascading are usually optimized for single-pass cost and quality, treating user response as exogenous, even though users re-prompt or abandon after failures, which affects provider revenue and user welfare.

Approach

Models multi-round routing as a Stackelberg game with a Markov model of user retries and abandonment; fully characterizes the user best response (threshold rules on net value xi_i = V p_i - t_i) and reduces the provider problem to a single-variable optimization, showing static routing is almost always optimal.

Key results

• Provider-optimal routing is almost always static (no cascade) across parameter regimes; cascading is optimal only in a narrow regime where models differ in net value.
• Misalignment between provider and user arises when cost-of-pass rankings conflict with user value rankings; the provider may route to the weaker standard model even when the reasoning model is preferred by users.
• If churn penalty P is at most min{c_1/p_1, c_2/p_2}, throttling latency above V p_i lowers provider cost (J_post <= J_pre).

Surprising findings

• Even though the reasoning model is preferred by users, the provider-optimal policy routes to the standard model unless its cost-of-pass is sufficiently high.
• Providers are incentivized to throttle latency when the churn penalty is low, deteriorating user utility.

Limitations

• Only two models and a single provider and single user are considered.
• Assumes i.i.d. success probabilities, fixed known user value V and provider penalty P, and that users observe the routing strategy.
• Subscription framing abstracts away per-query monetary prices.

Experimental setup

Metrics: user utility provider cost misalignment gap

Builds on: cost-of-pass

Closest papers by meaning

• Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• A Unified Approach to Routing and Cascading for LLMs (ICML 2025 · similarity 0.87)
• R2-Router: A New Paradigm for LLM Routing with Reasoning (ICML 2026 · similarity 0.86)
• Pricing Online LLM Services with Data-Calibrated Stackelberg Routing Game (AAAI 2026 · similarity 0.86)
• Causal LLM Routing: End-to-End Regret Minimization from Observational Data (NeurIPS 2025 · similarity 0.86)
• BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute (ICML 2025 · similarity 0.86)