Research map / Large language models / Reasoning and chain of thought
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a BERT-style (DeBERTa-v3-large) router that sends easy queries to a small LLM and hard ones to a large LLM, using probabilistic and transformed soft labels from BART-score quality gaps, to trade quality for cost at test time.
Problem
Serving every query with an expensive large LLM is costly, while small LLMs lag in response quality; a method is needed to route queries between small and large models with a tunable quality-cost tradeoff.
Approach
Train an encoder router to predict the probability that the small model's response is at least as good as the large model's (using sampled responses and BART-score quality gaps), with a data transformation that relaxes the threshold by t to fix label imbalance when the small model is much weaker; a single model call is made per query.
Key results
- With Llama-2 (13b) as large and GPT-3.5-turbo as the small-large pair, the router achieves 22% cost advantage with about 1% BART score drop, and up to 40% fewer large-model calls with no quality drop.
- For Llama-2 (7b) vs Llama-2 (13b), the router with r_det routes 20% of queries to the small model with 0.1% quality drop and 40% with 0.2% drop.
- For the large performance gap (FLAN-t5 800m vs Llama-2 13b), r_trans achieves 40% cost advantage with 10.3% quality drop, 3.5% lower than r_det.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add probabilistic router training (soft labels from 10 sampled responses) instead of deterministic labels | router trained on Llama-2 (7b)/Llama-2 (13b) pair · MixInstruct small gap, 20% cost advantage | BART score drop (%): 0.1 -> -0.1 | Helped |
| add data transformation (relaxed threshold t* labels) to probabilistic router (ablation) | router on FLAN-t5 (800m)/Llama-2 (13b) large gap pair · MixInstruct large gap, 40% cost advantage | BART score drop (%): 13.1 -> 10.3 vs r_prob | Helped |
| add data transformation to probabilistic router | router on Llama-2 (13b)/GPT-3.5-turbo pair · MixInstruct medium gap, 20% cost advantage | BART score drop (%): 0.8 vs 0.8 | No clear effect |
| add data transformation to probabilistic router at low cost advantage | router on Llama-2 (7b)/Llama-2 (13b) small gap pair · MixInstruct small gap, 10% cost advantage | BART score drop (%): -0.1 (r_prob) vs 0.1 (r_trans) | Hurt |
| probabilistic router vs deterministic router on medium gap pair at 40% cost advantage | router on Llama-2 (13b)/GPT-3.5-turbo · MixInstruct medium gap, 40% cost advantage | BART score drop (%): r_prob slightly outperforms r_det | Helped |
| route queries to small model via router instead of all-at-large | Llama-2 (7b) routed with Llama-2 (13b) at 20% and 40% cost advantage · MixInstruct small gap | BART score drop (%): up to 20% cost advantage with no drop (r_prob, r_trans) | Helped |
| router-based routing with large quality gap (FLAN-t5 800m vs Llama-2 13b) | r_det and r_prob routers · MixInstruct large gap, 40% cost advantage | BART score drop (%): marginally better than random | No clear effect |
| use BART-score-trained routers evaluated with GPT-4 score under high quality-gap correlation | routers trained on BART quality gaps · GPT-4 evaluation, high correlation (r=0.46) | GPT-4 score performance drop: up to 1% drop at 20% cost advantage, 2.1% at 40% | Helped |
| router DeBERTa-v3-large (300M) instead of LLM generation for routing decision | routing overhead · latency measurement | latency (s): 0.036 s vs 0.46 s (FLAN-t5 800m) and 14.61 s (Llama-2 13b) | Helped |
Surprising findings
- Probabilistic router can beat the all-at-large baseline (negative quality drop) at modest cost advantages when the performance gap is small.
- Even at temperature 0, GPT-4 returns different responses, motivating non-deterministic labels.
Limitations
- Router is trained and tested on a fixed model pair and data distribution; generalization depends on quality-gap correlation.
- Relies on BART score as proxy for quality; correlation with GPT-4 scores varies by pair.
- Grid search for transformation parameter t is heuristic; large-gap pairs still show notable quality drop.
Experimental setup
Datasets: MixInstruct
Benchmarks: MixInstruct
Metrics: BART score cost advantage performance drop GPT-4 score
Baselines: all-at-large all-at-small random routing
Builds on: DeBERTa-v3 BART score MixInstruct
Closest papers by meaning
- BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute (ICML 2025 · similarity 0.91)
- Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving (NeurIPS 2025 · similarity 0.89)
- MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees (NeurIPS 2025 · similarity 0.89)
- Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning (AAAI 2026 · similarity 0.89)
- Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.89)
- Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.