Research map / Large language models / Reasoning and chain of thought
Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Router-R1 trains an LLM router with PPO-style RL to interleave think steps with multi-round routing to a pool of LLMs, using format, exact-match, and cost rewards.
Problem
Existing LLM routers make single-round, one-to-one model assignments, which limits their ability to solve complex multi-hop tasks needing complementary strengths of multiple LLMs, and they lack flexible performance-cost control.
Approach
Treat multi-LLM routing and aggregation as a sequential decision process where the router is itself a reasoning LLM that alternates think actions and route actions to pool models, trained with RL using format, outcome (EM), and cost rewards; it conditions only on textual model descriptors so new LLMs can be added without retraining.
Key results
- Router-R1-Qwen reaches 0.416 average EM across seven QA datasets, above the best baseline Largest LLM at 0.338 with Qwen2.5-3B base.
- Router-R1-Llama reaches 0.409 average EM, versus Search-R1 at 0.311 and Prompt LLM at 0.346 for the Llama backbone.
- With cost coefficient alpha=0.9, Router-R1-Qwen cost drops to about 5.5 versus 150.6 at alpha=0.0 on NQ, with EM falling from 0.388 to 0.244.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-round Router-R1 routing instead of the strongest listed baseline | Qwen2.5-3B-Instruct · Seven QA benchmarks, Avg. | Exact Match: 0.291 -> 0.416 | Helped |
| Use multi-round Router-R1 routing instead of the strongest listed baseline | Llama-3.2-3B-Instruct · Seven QA benchmarks, Avg. | Exact Match: 0.350 -> 0.409 | Helped |
| Add Palmyra-Creative-122B and LLaMA3-ChatQA-1.5-8B at inference (ablation) | Router-R1-Qwen · NQ, TriviaQA, PopQA, HotpotQA | Exact Match: 0.458 -> 0.463 | Mixed |
| Compress model descriptors (ablation) | Router-R1-Qwen · Seven QA benchmarks, Avg. | Exact Match: 0.416 -> 0.419 | No clear effect |
| Add LLaMA3-ChatQA-1.5-8B during training (ablation) | Router-R1-Qwen · Seven QA benchmarks, Avg. | Exact Match: 0.416 -> 0.425 | Helped |
| Increase the cost coefficient from 0.0 to 0.9 (ablation) | Router-R1-Qwen · NQ | Cost: 150.6 -> 5.5 | Helped |
| Increase the cost coefficient from 0.0 to 0.9 (ablation) | Router-R1-Qwen · NQ | Exact Match: 0.388 -> 0.244 | Hurt |
| Increase the cost coefficient from 0.0 to 0.6 (ablation) | Router-R1-Qwen · PopQA | Exact Match: 0.384 -> 0.414 | Helped |
| Increase the cost coefficient from 0.0 to 0.6 (ablation) | Router-R1-Qwen · PopQA | Cost: 98.3 -> 75.9 | Helped |
Surprising findings
- Adding a new unseen LLM at inference with only textual descriptors slightly improved average EM (0.416 to 0.419 in the sensitivity study and new best on several datasets).
Limitations
- Experiments limited to QA tasks; cost normalization may not transfer to heterogeneous domains like math or code.
- Cost reward is based on output token count and a predefined pricing function, not measured latency.
- Evaluation uses random 500-instance subsets per dataset rather than full test sets.
Experimental setup
Datasets: Natural Questions HotpotQA
Benchmarks: NQ TriviaQA PopQA HotpotQA 2WikiMultiHopQA Musique Bamboogle
Metrics: exact match F1 cost
Baselines: Direct CoT SFT RAG Search-R1 Prompt LLM Largest LLM KNN Router MLP Router BERT Router RouterDC GraphRouter
Builds on: PPO GRPO Search-R1 RouteLLM FrugalGPT
Closest papers by meaning
- RouteLLM: Learning to Route LLMs from Preference Data (ICLR 2025 · similarity 0.91)
- R2-Router: A New Paradigm for LLM Routing with Reasoning (ICML 2026 · similarity 0.90)
- Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning (ICML 2026 · similarity 0.89)
- Causal LLM Routing: End-to-End Regret Minimization from Observational Data (NeurIPS 2025 · similarity 0.88)
- Routing, Cascades, and User Choice for LLMs (ICLR 2026 · similarity 0.88)
- DiSRouter: Distributed Self-Routing for LLM Selections (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.