Research map / Large language models / Reasoning and chain of thought
AutoMix: Automatically Mixing Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoMix, which routes queries from a small black-box LM to a larger LM using few-shot self-verification (entailment-style) and a POMDP-based router, evaluated on five context-grounded reasoning datasets.
Problem
Choosing which LLM of varying size and cost to query for each input under black-box API access, without large task-specific routing training data, while balancing cost and performance.
Approach
Few-shot self-verification framed as context-grounded entailment by the small model itself, with a POMDP router that treats noisy verifier probabilities as observations of query difficulty and decides whether to accept the answer or route to a larger model; it also supports unsolvable-query detection and needs as few as 50 training examples.
Key results
- AutoMix+POMDP achieves the largest Delta-IBC on most datasets, e.g., 156.8 on DIPLOMAT and 51.6 on QuALITY with Mistral-7B as SLM versus FrugalGPT's 16.8 and 8.1.
- With GPT-3.5 as SLM, AutoMix+POMDP reaches 114.7 Delta-IBC on QASPER versus FrugalGPT's 87.2.
- Abstract claims over 50% computational cost reduction for comparable performance; with 50 training examples AutoMix keeps about 15% Delta-IBC and beats baselines by over 20 absolute points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the POMDP router instead of FrugalGPT or HybridLLM routing | AutoMix with Mistral-7B as the small model · Diplomat, Mutual, CoQA, QASPER, and QuaLITY | ΔIBC (%) | Helped |
| Use the POMDP router instead of FrugalGPT or HybridLLM routing | AutoMix with Llama2-13B as the small model · Diplomat, Mutual, CoQA, and QuaLITY | ΔIBC (%) | Helped |
| Use the POMDP router instead of FrugalGPT routing | AutoMix with Llama2-13B as the small model · QASPER | ΔIBC (%): 8.5 vs 9.4 | No clear effect |
| Use the POMDP router instead of FrugalGPT or HybridLLM routing | AutoMix with GPT-3.5 as the small model · Diplomat, Mutual, CoQA, QASPER, and QuaLITY | ΔIBC (%) | Helped |
| Replace the threshold router with the POMDP router (ablation) | AutoMix · Across the five datasets and three small models | ΔIBC (%) | Mixed |
Surprising findings
- Few-shot self-verification identifies correct answers well enough to be useful for routing even though self-correction is generally unreliable.
- Chained AutoMix (two-stage routing) underperforms a single-stage AutoMix in the three-model setting, since it cannot skip directly from SLM to LLM.
Limitations
- Designed for context-grounded reasoning and dialogue tasks; may not transfer to factual QA or commonsense reasoning.
- Cost model relies on assumed relative API price ratios (1:60 to 1:200); verifier cost set equal to SLM cost.
- Results depend on the specific models and datasets; baselines' router cost assigned zero at inference.
Experimental setup
Datasets: QASPER QuALITY CoQA MuTual DIPLOMAT
Benchmarks: QASPER QuALITY CoQA MuTual DIPLOMAT
Metrics: IBC lift (Delta IBC) F1 accuracy cost-performance curves
Baselines: FrugalGPT HybridLLM Chained AutoMix Union AutoMix
Builds on: FrugalGPT HybridLLM AdaOps AdaptiveConsistency
Closest papers by meaning
- Gorilla: Large Language Model Connected with Massive APIs (NeurIPS 2024 · similarity 0.87)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.86)
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (ICLR 2024 · similarity 0.86)
- Adaptive Draft-Verification for Efficient Large Language Model Decoding (AAAI 2025 · similarity 0.85)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.85)
- Fast Inference for Augmented Large Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.