Research map / Large language models / Reasoning and chain of thought

BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

ICML 2025 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BEST-Route, a multi-head router that jointly picks a small LLM and a best-of-n sample count per query, using a proxy reward model for cheap best-of-n selection, to cut LLM serving cost by up to 60% with under 1% quality drop.

Problem

LLM query routing methods usually send one response per query and so overuse expensive large models; small models with multiple samples could match large-model quality more cheaply.

Approach

A shared DeBERTa backbone with K×N lightweight heads predicts, for each small model and best-of-n count, the probability that its proxy-selected best-of-n response matches GPT-4o; the router picks the cheapest combination above a threshold, and a small DeBERTa proxy reward model performs the best-of-n selection.

Key results

• BEST-Route achieves 60% cost reduction with 0.80% armoRM quality drop vs always using GPT-4o, versus 5.08% for N-label routing.
• On MT-Bench OOD, BEST-Route reaches 60% cost reduction with 1.59% quality drop, up to 4.3% better than the strongest baseline.
• On coding queries, after adding Codestral-22B, BEST-Route achieves 20% cost reduction while slightly exceeding GPT-4o quality (-0.50% drop).

Measured effects reported in the paper

Surprising findings

• On coding queries, routing with best-of-n small models achieved response quality slightly above always using GPT-4o at 10-20% cost reduction.
• N-class and clustering routers achieve almost no cost reduction because they default to the reference model.

Limitations

• Relies on proxy reward model accuracy for best-of-n selection, which may misrank responses.
• Scalability to very large model pools (hundreds of LLMs) is unexplored.
• Ground-truth labels and evaluation rely on armoRM, a single automatic judge.

Experimental setup

Datasets: MixInstruct RewardBench CodeUltraFeedback BeaverTails MT-Bench

Benchmarks: MixInstruct-based mixed test set (1K) MT-Bench (OOD)

Metrics: armoRM quality drop vs GPT-4o (%) cost reduction (%) BLEU drop ROUGE drop router latency cost estimation error

Baselines: N-class routing N-label routing Clustering-based routing Model cascades (FrugalGPT-style) Best-of-n per single LLM

Builds on: best-of-n sampling RouteLLM FrugalGPT armoRM OpenAssistant reward model

Closest papers by meaning

• Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (ICLR 2024 · similarity 0.91)
• Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.91)
• Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving (NeurIPS 2025 · similarity 0.91)
• A Unified Approach to Routing and Cascading for LLMs (ICML 2025 · similarity 0.90)
• Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.89)
• Causal LLM Routing: End-to-End Regret Minimization from Observational Data (NeurIPS 2025 · similarity 0.89)