AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Empowering LLM Tool Invocation with Tool-call Reward Model

Da Ma, Ziyue Yang, Hongshen Xu, Haotian Fang, Kai Yu, Lu Chen

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a tool-call reward model to provide fine-grained credit assignment for RL-based LLM tool invocation, beyond outcome-only rewards.

Problem

Existing agentic RL approaches for LLM tool use rely on outcome-only reward signals that assign credit at a coarse granularity and often induce poor tool-call behavior.

Approach

A reward model that scores individual tool calls, giving step-level (tool-call-level) credit signals during RL training of tool-invoking LLMs, instead of relying solely on final outcome rewards.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add TRM per-tool-call reward to PPOSearch-R1-PPO · Search QA averageavg exact match: 32.75 -> 34.93 (3B); 37.94 -> 39.95 (7B)Improved
add TRM per-tool-call reward to GRPOSearch-R1-GRPO · Search QA averageavg exact match: 42.33 -> 43.49 (3B); 46.90 -> 48.62 (7B)Improved
add TRM reward to ToRLToRL-PPO and ToRL-GRPO · Math average (5 benchmarks)avg accuracy: 43.18 -> 45.42 (1.5B GRPO); 52.19 -> 53.70 (7B GRPO)Improved
add TRM rewardSearch-R1-GRPO 7B · 2WikiMultiHopQAexact match: 47.80 -> 47.67No clear effect
add TRM rewardToRL-PPO 1.5B · AMC23accuracy: 55.00 -> 50.00Hurt
use turn-level instead of group-level advantage estimation (ablation)GRPO with TRM · HotpotQA / search QAaccuracy: 41.18 -> 42.47Improved
use TRM per-call scoring versus trajectory-level ORM (ablation)PPO with Qwen2.5-3B · search QAaccuracyImproved
use both necessity and quality scores instead of quality-only (ablation)TRM · search QAaccuracyImproved
use necessity-only scores (ablation)TRM · search QAaccuracyHurt
increase TRM weight alpha (ablation)PPO and GRPO with TRM · search QAaccuracyMixed
add TRM during training on searchRL-trained LLM · math with Python code (transfer)accuracyImproved
use TRM instead of StepSearch or AgentPRM supervision (ablation)PPO with Qwen2.5-3B · search QAaccuracyImproved

Experimental setup

Baselines: outcome-only reward RL

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy