Research map / Large language models / Instruction and fine-tuning
ToolRL: Reward is All Tool Learning Needs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies reward design for RL-based tool learning and proposes a fine-grained rule-based reward (format + tool-call correctness) trained with GRPO from cold start on Qwen2.5 and Llama-3.2 instruct models, outperforming SFT and PPO on BFCL, API-Bank, and Bamboogle.
Problem
Training LLMs for general-purpose tool selection and application with RL is hampered by reward design: multiple tools with many parameters make coarse answer-matching rewards uninformative, and SFT generalizes poorly to unfamiliar or complex tool use.
Approach
A rule-based reward combining a binary format reward with a fine-grained correctness reward that matches tool names, parameter names and parameter values via optimal matching between predicted and ground-truth tool calls, scaled to [-3, 3], used in GRPO without a KL term; ablations on length, scale, granularity, and temporal dynamics guide the design.
Key results
- Qwen2.5-7B-Instruct with GRPO cold start reaches 58.38% overall on BFCL V3 versus 41.97% for the raw model and 36.53% for SFT4k.
- On API-Bank, Qwen2.5-1.5B GRPO cold start achieves 63.15% overall, versus 30.65% raw and 47.07% for SFT4k.
- On Bamboogle, Qwen2.5-7B raw model scores 69.6% and GRPO cold start 72.0%, with GRPO cold start the best among trained models for 7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with GRPO and the ToolRL reward from a cold start rather than the strongest listed baseline | Qwen2.5-1.5B-Instruct · BFCL V3 | Overall Acc: 42.95% -> 46.20% | Helped |
| train with GRPO and the ToolRL reward from a cold start rather than PPO cold start | Qwen2.5-3B-Instruct · BFCL V3 | Overall Acc: 51.15% -> 52.98% | Helped |
| train with GRPO and the ToolRL reward from a cold start rather than PPO cold start | Qwen2.5-7B-Instruct · BFCL V3 | Overall Acc: 46.68% -> 58.38% | Helped |
| train with GRPO and the ToolRL reward from a cold start rather than SFT4k | Llama-3.2-3B-Instruct · BFCL V3 | Overall Acc: 44.16% -> 44.10% | No clear effect |
| train with GRPO and the ToolRL reward from a cold start rather than the strongest listed baseline | Qwen2.5-1.5B-Instruct · API-Bank Test | Overall Acc: 61.31% -> 63.15% | Helped |
| train with GRPO and the ToolRL reward from a cold start rather than PPO cold start | Qwen2.5-3B-Instruct · API-Bank Test | Overall Acc: 57.62% -> 67.00% | Helped |
| train with GRPO and the ToolRL reward from a cold start rather than the raw model | Qwen2.5-7B-Instruct · API-Bank Test | Overall Acc: 62.48% -> 64.66% | Helped |
| train with GRPO and the ToolRL reward from a cold start rather than SFT400+PPO | Llama-3.2-3B-Instruct · API-Bank Test | Overall Acc: 57.79% -> 59.13% | Helped |
| add a reasoning-length reward (ablation) | ToolRL GRPO · BFCL V3 | Overall Acc: 52.98% -> 48.89% | Hurt |
| use a dynamic reasoning-length reward (ablation) | ToolRL GRPO · BFCL V3 | Overall Acc: 52.98% -> 48.24% | Hurt |
| equalize the maximum correctness and format rewards (ablation) | ToolRL GRPO · BFCL V3 | Overall Acc: 52.98% -> 51.76% | Hurt |
| use two-stage reward scaling (ablation) | ToolRL GRPO · BFCL V3 | Overall Acc: 52.98% -> 50.66% | Hurt |
Surprising findings
- Adding a length reward or dynamic length reward does not improve and can substantially degrade performance for smaller models (e.g., Qwen2.5-1.5B drops from 46.20% to 33.23% on BFCL).
- Two-stage coarse reward scale switching reduced performance, while smooth dynamic scaling helped.
Limitations
- Llama-3.2 models show smaller gains from GRPO, attributed possibly to lower adaptability.
- Performance of GRPO on multi-turn BFCL remains very low (e.g., under 5% for most models).
- Results are reported on a single run configuration with 4K training examples; scaling beyond 4K gave no noticeable gains.
Experimental setup
Datasets: ToolACE Hammer (Masked) xLAM
Benchmarks: BFCL V3 API-Bank Bamboogle
Metrics: accuracy average number of tool calls
Baselines: Raw Instruct Model SFT on RL data (400 and 4K) SFT400+GRPO PPO Cold Start SFT400+PPO
Builds on: GRPO PPO DeepSeek-R1-style rule-based rewards veRL
Closest papers by meaning
- ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.93)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.91)
- Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning (ICLR 2026 · similarity 0.91)
- Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.90)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.89)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.