Research map / Large language models / Reasoning and chain of thought
Routing and Reasoned Evaluation with Large Language Models
Guiyao Tie, Tianyao Luo, Xueyang Zhou, Chaoran Hu, Yunhong He, Junran Wu, Yuanfan Yao, Pan Zhou, Lichao Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R$^2$Eval, a routing-aware framework for LLM-as-judge evaluation that allocates evaluation resources across models to address heterogeneous reliability, latency, and token cost.
Problem
LLM-based automated assessment of model outputs faces heterogeneous reliability across judge models, high latency and token costs, and no principled way to allocate evaluation resources.
Approach
A routing-aware evaluation framework (R$^2$Eval) that routes evaluation requests across LLM judges with reasoned evaluation, aiming to balance reliability against cost and latency. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| route judges with offline ACC-only policy instead of fixed GPT5 | judge selection · R2Eval routing (average over domains) | accuracy (%): 66.0 -> 75.4 | Improved |
| route judges with offline ACC-only policy instead of fixed GPT5 | judge selection · R2Eval routing (average over domains) | selected-judge time (s): 41.5 -> 18.7 | Improved |
| use ACC-Time policy instead of ACC-only | offline routing · R2Eval routing (average over domains) | accuracy (%) / time (s): 75.4 -> 71.9 accuracy; 18.7 -> 8.1 time | Mixed |
| add cost penalty (ACC-Time-Cost) to offline routing | offline routing · R2Eval routing (average over domains) | accuracy (%) / token cost: 74.1 acc, 2.7 cost vs ACC-only 75.4, 3.0 | Mixed |
| use template-only Router 1 for online routing | online routing · R2Eval routing (average over domains) | accuracy (%): 66.9 vs ACC-Time 71.9 | Hurt |
| use trained Router 3 (Qwen2.5-7B) over ACC-Time | online routing · R2Eval Coding and Math | accuracy (%): Coding 63.0 -> 74.6 | Improved |
| use Llama-based Router 2 | online routing · R2Eval Writing | accuracy (%): 77.2 | Improved |
| use trained routers instead of ACC-only | online routing · R2Eval routing (average over domains) | selected-judge latency: 20-45% lower | Improved |
| use LRM judges instead of LLM judges on reasoning domains | judges · R2Eval math and logic | accuracy (%): Gemini-2.5-Flash 74.90 reasoning avg | Improved |
| use explicit multi-step reasoning on subjective domains | LRM judges · R2Eval non-reasoning | accuracy (%) | No clear effect |
| longer thinking length | LRM judges · R2Eval reasoning tasks | accuracy correlation: r = 0.28 | Mixed |
| use mid-sized Qwen3-32B instead of strongest LRMs | judges · R2Eval | latency and accuracy: latency -35-50%, accuracy within 2-3 pp | Improved |
Closest papers by meaning
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.87)
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.86)
- Dynamic Evaluation of Large Language Models by Meta Probing Agents (ICML 2024 · similarity 0.86)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.86)
- Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.86)
- Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
