Research map / Large language models / Reasoning and chain of thought

RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RACER, a post-hoc, model-agnostic method that calibrates a threshold over nested LLM sets (with a null model for abstention) to control misrouting risk at level alpha, then aggregates outputs of the selected models to improve accuracy.

Problem

Single-model LLM routers misrank candidates and misroute queries; existing subset-routing uses heuristic size controls without coverage guarantees. The paper asks how to bound the selected set size while guaranteeing it contains a correct model.

Approach

Formulates LLM routing as alpha-VOR (minimize expected set size subject to misrouting risk <= alpha), builds nested model sets by thresholding augmented non-conformity scores (including a virtual null model for abstention), and calibrates the threshold with conformal risk control so that the guarantee holds distribution-free on unseen queries.

Key results

• RACER-P with RouterDC achieves 62.1% average accuracy across four benchmarks, versus 57.1% for the best single candidate LLM (+5.0%).
• RACER improves MLPR by up to 4.0% on a single benchmark (C-MMLU) and 3.6% on average.
• Relative to full-model aggregation, RACER reduces model calls by up to 58.6% while gaining up to 4.49% accuracy.

Surprising findings

• Invoking all candidate models and aggregating can hurt accuracy; excluding models via RACER gives higher accuracy than full ensemble in most dataset-router pairs.

Limitations

• Guarantee relies on exchangeability between calibration and test data.
• Ground-truth sets require labeled correctness of all candidate LLM outputs on calibration data.
• Aggregation hyperparameters (alpha, aggregation method) are tuned on validation data and the confidence-based weights use a single unified evaluator model.

Experimental setup

Datasets: GSM8K MMLU CMMLU ARC-Challenge

Benchmarks: GSM8K MMLU CMMLU ARC-Challenge

Metrics: misrouting risk expected set size accuracy

Baselines: RouterDC MLPR KNNR full model aggregation best single candidate LLM

Builds on: Conformal Risk Control RouterDC MLPR KNNR

Closest papers by meaning

• Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.90)
• Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning (AAAI 2026 · similarity 0.90)
• BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute (ICML 2025 · similarity 0.89)
• Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.89)
• Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.89)
• A Unified Approach to Routing and Cascading for LLMs (ICML 2025 · similarity 0.88)