Research map / Large language models / Reasoning and chain of thought
Empowering LLM Tool Invocation with Tool-call Reward Model
Da Ma, Ziyue Yang, Hongshen Xu, Haotian Fang, Kai Yu, Lu Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a tool-call reward model to provide fine-grained credit assignment for RL-based LLM tool invocation, beyond outcome-only rewards.
Problem
Existing agentic RL approaches for LLM tool use rely on outcome-only reward signals that assign credit at a coarse granularity and often induce poor tool-call behavior.
Approach
A reward model that scores individual tool calls, giving step-level (tool-call-level) credit signals during RL training of tool-invoking LLMs, instead of relying solely on final outcome rewards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TRM per-tool-call reward to PPO | Search-R1-PPO · Search QA average | avg exact match: 32.75 -> 34.93 (3B); 37.94 -> 39.95 (7B) | Improved |
| add TRM per-tool-call reward to GRPO | Search-R1-GRPO · Search QA average | avg exact match: 42.33 -> 43.49 (3B); 46.90 -> 48.62 (7B) | Improved |
| add TRM reward to ToRL | ToRL-PPO and ToRL-GRPO · Math average (5 benchmarks) | avg accuracy: 43.18 -> 45.42 (1.5B GRPO); 52.19 -> 53.70 (7B GRPO) | Improved |
| add TRM reward | Search-R1-GRPO 7B · 2WikiMultiHopQA | exact match: 47.80 -> 47.67 | No clear effect |
| add TRM reward | ToRL-PPO 1.5B · AMC23 | accuracy: 55.00 -> 50.00 | Hurt |
| use turn-level instead of group-level advantage estimation (ablation) | GRPO with TRM · HotpotQA / search QA | accuracy: 41.18 -> 42.47 | Improved |
| use TRM per-call scoring versus trajectory-level ORM (ablation) | PPO with Qwen2.5-3B · search QA | accuracy | Improved |
| use both necessity and quality scores instead of quality-only (ablation) | TRM · search QA | accuracy | Improved |
| use necessity-only scores (ablation) | TRM · search QA | accuracy | Hurt |
| increase TRM weight alpha (ablation) | PPO and GRPO with TRM · search QA | accuracy | Mixed |
| add TRM during training on search | RL-trained LLM · math with Python code (transfer) | accuracy | Improved |
| use TRM instead of StepSearch or AgentPRM supervision (ablation) | PPO with Qwen2.5-3B · search QA | accuracy | Improved |
Experimental setup
Baselines: outcome-only reward RL
Closest papers by meaning
- ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.91)
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.90)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.90)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.90)
- ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (ICLR 2026 · similarity 0.89)
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
