Research map / Large language models / Reasoning and chain of thought

Universal Model Routing for Efficient LLM Inference

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniRoute, a model routing method that represents each LLM by its per-cluster (or learned-cluster) prediction-error vector on a small validation set, enabling routing to unseen LLMs at test time without retraining.

Problem

Model routing is usually trained for a fixed pool of LLMs, so new or deprecated models require costly router retraining; the paper targets routing over dynamic pools that include LLMs unseen during training.

Approach

Represent each LLM as a feature vector of its prediction errors on representative prompts (per-cluster average errors on a small validation set), so any new LLM can be routed to by cost-adjusted expected error on the prompt's cluster; the cluster assignment is either unsupervised k-means or a learned softmax map. Shows K-NN routing is a special case and derives an excess risk bound.

Key results

• On EmbedLLM with more than 30 unseen LLMs, UniRoute gives gains over K-NN on all metrics, with statistically significant improvements.
• UniRoute consistently outperforms ZeroRouter across all four evaluated datasets.
• UniRoute is often significantly better than K-NN across validation sample sizes, since it can exploit training data.

Surprising findings

• Most existing learned routers (MLP, matrix factorization, BERT) are inadmissible for dynamic pools because their outputs are tied to a fixed set of training LLMs.

Limitations

• Not guaranteed to recover the performance of a static linear router in the fully static pool setting, since it depends on the validation sample.
• Robustness to prompt distribution shift is not addressed; the design space of instantiations is only partially explored.
• Evaluation is limited to binary accuracy metrics.

Experimental setup

Datasets: EmbedLLM RouterBench SPROUT o3-mini Math+Code (Minerva Math, LiveCodeBench subset) Chatbot Arena

Benchmarks: EmbedLLM RouterBench SPROUT o3-mini Math+Code Chatbot Arena

Metrics: area under deferral curve area up to 50% cost quality-neutral cost

Baselines: ZeroRouter K-NN router MLP router (clairvoyant) matrix factorization router (clairvoyant)

Builds on: K-NN router ZeroRouter RouteLLM learning to defer

Closest papers by meaning

• RouteLLM: Learning to Route LLMs from Preference Data (ICLR 2025 · similarity 0.92)
• Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning (AAAI 2026 · similarity 0.91)
• Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.91)
• Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.91)
• RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models (ICML 2026 · similarity 0.90)
• Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.90)