Research map / Large language models / Reasoning and chain of thought
RouteLLM: Learning to Route LLMs from Preference Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a framework for training binary LLM routers (strong vs. weak model) from human preference data (Chatbot Arena) augmented with MMLU golden labels and GPT-4-judged Nectar data, reducing GPT-4 calls by over 2x while keeping quality.
Problem
Choosing between an expensive strong LLM and a cheap weak LLM per query trades off cost against response quality; routers must generalize out-of-domain and across model pairs without retraining.
Approach
Learn a win-prediction model from human preference labels (Chatbot Arena, tiered models) and augment with golden-labeled and LLM-judge-labeled data; compare four router architectures (similarity-weighted Bradley-Terry ranking, matrix factorization, BERT classifier, Llama 3 8B causal classifier) with a threshold for cost-quality control and new metrics (CPT, APGR).
Key results
- On MT Bench with Arena+Judge data, matrix factorization reaches CPT(80%) of 31.31% and APGR 0.802 vs. random's 0.500, cutting GPT-4 calls substantially; cost savings up to 3.66x at 95% GPT-4 quality.
- On MMLU, augmenting Arena data with MMLU golden labels lets all routers need ~20% fewer GPT-4 calls than random for CPT(50%), using only ~1.5k extra samples.
- Routers trained on Arena+Judge data transfer to unseen model pairs (Claude 3 Opus/Sonnet, Llama 3.1 70B/8B) without retraining, with APGR up to 0.772 and 0.767.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GPT-4-judge preference data to Arena training | LLM routers · MT Bench | APGR: 0.391 -> 0.751 (BERT); 0.573 -> 0.679 (Causal LLM); 0.580 -> 0.802 (Matrix Factorization); 0.610 -> 0.759 (SW Ranking) | Helped |
| Add golden-labeled MMLU data to Arena training | LLM routers · MMLU 5-shot | APGR: 0.502 -> 0.572 (BERT); 0.499 -> 0.600 (Causal LLM); 0.524 -> 0.597 (Matrix Factorization); 0.473 -> 0.603 (SW Ranking) | Helped |
| Add GPT-4-judge preference data to Arena training | LLM routers · GSM8K 8-shot | APGR: 0.438 -> 0.531 (BERT); 0.461 -> 0.622 (Causal LLM); 0.4746 -> 0.565 (Matrix Factorization); 0.4753 -> 0.568 (SW Ranking) | Helped |
| Train BERT router on Arena preferences rather than using the random router | BERT router · MT Bench | APGR: 0.500($\pm$0.02) -> 0.391 | Hurt |
| Train routers on Arena preferences without benchmark-specific augmentation | LLM routers · MMLU 5-shot | APGR: 0.500($\pm$0) baseline; Arena routers: 0.473–0.524 | No clear effect |
| Train routers on Arena preferences without benchmark-specific augmentation | LLM routers · GSM8K 8-shot | APGR: 0.497($\pm$0.01) baseline; Arena routers: 0.438, 0.461, 0.4746, 0.4753 | Hurt |
| Use the trained router on Claude 3 Opus / Claude 3 Sonnet without retraining | LLM routers · MT Bench | APGR: 0.493 ($\pm$0.033) baseline; router values 0.656–0.772 | Helped |
| Use the trained router on Llama 3.1 70B / Llama 3.1 8B without retraining | LLM routers · MT Bench | APGR: 0.512 ($\pm$0.017) baseline; router values 0.673–0.767 | Helped |
Surprising findings
- Routers trained only on Chatbot Arena data (BERT and causal LLM) perform near or below random on MT Bench and MMLU, attributed to high-capacity models overfitting in a low-data regime.
- Matrix factorization and similarity-weighted ranking are competitive with or better than the 8B causal LLM and BERT classifiers despite being much simpler.
Limitations
- Evaluated only on a two-model binary routing setting (GPT-4 vs Mixtral-8x7B) and public benchmarks that may differ from real workloads.
- Performance varies widely between routers trained on the same data without a clear explanation.
- Arena-only training transfers poorly to out-of-distribution benchmarks such as MMLU and GSM8K, requiring in-domain augmentation.
Experimental setup
Datasets: Chatbot Arena 80k battles MMLU validation split Nectar MT Bench MMLU GSM8K
Benchmarks: MT Bench MMLU GSM8K
Metrics: CPT(50%) CPT(80%) APGR PGR cost saving ratio router overhead cost
Baselines: Random router BERT classifier Unify AI Martian Hybrid-LLM Zooter FrugalGPT AutoMix LLM-Blender
Builds on: Bradley-Terry model Hybrid-LLM Chatbot Arena Nectar dataset
Closest papers by meaning
- Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.92)
- GraphRouter: A Graph-based Router for LLM Selections (ICLR 2025 · similarity 0.91)
- Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning (AAAI 2026 · similarity 0.91)
- Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.90)
- Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.