Research map / Large language models / Language, safety and interpretability
Smoothie: Label Free Language Model Routing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Smoothie performs label-free LLM routing by estimating sample-conditional LLM quality scores from embedding distances between LLM outputs, using a Gaussian weak-supervision model with kernel smoothing over nearest neighbors.
Problem
Choosing which LLM to route each input sample to when no labeled validation data is available, in multi-task settings where LLM quality varies across samples.
Approach
Adapts a weak-supervision multivariate Gaussian latent-variable model over SentenceBERT embeddings of (input, LLM output) pairs; closed-form estimates of each LLM's quality score come from pairwise embedding distances, and nearest-neighbor kernel smoothing makes the scores sample-conditional (Smoothie-Local).
Key results
- On NLG tasks, Smoothie-Global's learned weights correlate with ground-truth LLM performance at an average Spearman coefficient of 0.72 and identify the best LLM on 4/7 (3B) and 5/7 (7B) tasks.
- On mixed-task routing, Smoothie-Local beats baselines by up to 10 points accuracy over random routing and outperforms labeled baselines such as Labeled-KNN and PairRM, e.g., 58.7 vs 53.9 on the 3B mixed accuracy distribution.
- Smoothie-Train, which needs only held-out generations, matches Smoothie-Global on 12/14 model-dataset pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Smoothie global rather than random model selection | LLM selection · GSM8K | accuracy: 28.4 -> 37.5 | Helped |
| Use Smoothie global rather than random selection | LLM selection · MixInstruct | ChatGPT-rank: 5.96 -> 3.91 | Helped |
| Use sample-conditional local scoring instead of global scoring | Smoothie routing · Multi-task distributions | accuracy and ROUGE-2 | Helped |
| Estimate routing scores from held-out training examples rather than test-set generations (ablation) | Smoothie global · NLG tasks | task performance | Mixed |
| Estimate routing scores from held-out training examples, with neighborhood size 20 (ablation) | Smoothie local · Multi-task distributions | accuracy and ROUGE-2 | Hurt |
| Increase neighborhood size above 1 (ablation) | Smoothie local · Multi-task distributions | accuracy and ROUGE-2 | Hurt |
| Replace SentenceBERT embeddings with BGE-small embeddings (ablation) | Smoothie local · Multi-task distributions | accuracy and ROUGE-2: 58.7 -> 59.3; 20.2 -> 19.7; 75.0 -> 74.6; 26.9 -> 27.1 | Mixed |
| Use Smoothie global to select the best model from varied ensembles | Smoothie global · 350 NLG ensemble-task settings | best-model identification rate: 211 of them (60.2%) | Helped |
| Use Smoothie local rather than the best individual model on average | Multi-task routing · Mix-task distributions | generation quality | Helped |
Surprising findings
- Smoothie-Local with neighborhood size n0=1 performs best, and performance steadily degrades as n0 increases.
- Smoothie-Global with a single prompt template can approximately match a labeled best-on-validation baseline on several tasks despite using no labels.
Limitations
- Diagonal covariance assumption ignores dependencies between LLM error vectors.
- Optimizes only for quality and ignores cost tradeoffs between large and small models.
- Reliance on embeddings may capture only certain aspects of semantic similarity.
Experimental setup
Datasets: CNN/DailyMail XSum SQuAD TriviaQA E2E WebNLG LegalBench Definition Extraction AlpacaEval MixInstruct GSM8K
Benchmarks: CNN/DailyMail XSum SQuAD TriviaQA E2E WebNLG LegalBench Definition Extraction AlpacaEval MixInstruct GSM8K
Metrics: Rouge2 accuracy win-rate length-controlled win-rate ChatGPT-rank Spearman correlation solve rate
Baselines: Random Best-on-Val Labeled-KNN PairRM Best individual model
Builds on: weak supervision (Gaussian model) SentenceBERT
Closest papers by meaning
- Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.88)
- Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.87)
- GraphRouter: A Graph-based Router for LLM Selections (ICLR 2025 · similarity 0.87)
- RouteLLM: Learning to Route LLMs from Preference Data (ICLR 2025 · similarity 0.87)
- Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning (AAAI 2026 · similarity 0.87)
- BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.