Research map / Large language models / Reasoning and chain of thought

Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes STEER, a training-free, external-model-free step-level router that uses the small model's max-logit step confidence, modeled with a two-component GMM, to decide when to invoke the larger LLM.

Problem

Existing adaptive inference routers for LLM reasoning need trained external routers or PRMs and expensive data (e.g., Monte Carlo rollouts), and they are not robust under domain or model-family shifts.

Approach

Use the smaller model's own max-logit confidence per reasoning step, fit a two-component Gaussian mixture online to the step-confidence distribution across ongoing traces, and route the next step to the larger model when the posterior of the confident component falls below a threshold.

Key results

• On AIME, STEER reaches up to +20% accuracy with 48% fewer FLOPs than the larger model alone (Gemma3-4B/12B setting).
• With Gemma3-Instruct on math benchmarks, STEER averages 44.9% accuracy at 24.0e12 FLOPs, matching the 12B-only accuracy (44.9) at lower cost.
• On broader reasoning tasks with Qwen2.5-Instruct, STEER averages 45.5% accuracy with the lowest FLOPs (3.64 vs 5.20 for 7B-only), a 30% A/F gain over the baselines' best; within 1% of the top baseline accuracy.

Surprising findings

• RSD on Gemma3-Instruct math benchmarks yields lower accuracy than the larger model alone, attributed to PRM out-of-distribution issues.
• RSD sometimes incurs more FLOPs than running the larger model alone due to post-hoc step rejection invoking both models.

Limitations

• Relies on threshold grid search per benchmark for fairness, with tuned hyperparameters.
• Confidence distribution fitting assumes a bimodal structure and depends on sufficient ongoing traces per benchmark.
• Only evaluated with small (up to 12B) Gemma/Qwen models and a single larger-smaller pair per setting.

Experimental setup

Datasets: MATH500 OmniMath AIME 2022-2025 MuSiQue ACPBench KOR-Bench

Benchmarks: MATH500 OmniMath AIME ACPBench MuSiQue KOR-Bench

Metrics: accuracy FLOPs accuracy-per-FLOP

Baselines: SpecReason RSD Damani et al. (2024) query-level router 4B Only 12B Only 1.5B Only 7B Only

Builds on: RSD SpecReason speculative decoding

Closest papers by meaning

• Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.91)
• Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge (ICML 2026 · similarity 0.91)
• Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.91)
• RouteLLM: Learning to Route LLMs from Preference Data (ICLR 2025 · similarity 0.91)
• R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing (NeurIPS 2025 · similarity 0.91)
• TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks (ICLR 2026 · similarity 0.90)