Research map / Reinforcement learning / RL from feedback

ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ToolSearcher, an RL framework for large-scale tool selection over a 16k-tool repository, combining category-constrained tool discrimination, event-level search modeling, and trajectory-aligned credit allocation, evaluated on StableToolBench and AppWorld.

Problem

Tool selection from very large tool repositories under context-length limits requires multi-turn search, distinguishing similar tools, and composing compatible tools; existing agentic RL methods designed for knowledge QA do not handle this well.

Approach

Recasts tool search as category-constrained search to create hard discrimination training, gives event-level rewards for each search step that first discovers a target tool, and allocates selection and search rewards according to each trajectory's progress rather than a single outcome reward.

Key results

• On Qwen2.5-7B-Instruct, ToolSearcher reaches overall F1 0.513 on StableToolBench vs 0.496 for the second-best GDPO and 0.098 for multi-turn baseline.
• On Qwen3-4B-Instruct, overall F1 rises from 0.408 (multi-turn) to 0.531 on StableToolBench.
• On AppWorld with Qwen2.5-7B-Instruct, ToolSearcher gets avg TGC 0.334 vs 0.277 for GDPO.

Surprising findings

• Global search on Qwen2.5-7B gives higher F1 than the category-constrained setting (0.098 vs 0.088) despite CCTD being intended as a harder training environment.
• Removing event-level search modeling cuts average search rounds from about 4 to 2 and reduces SRecall sharply.

Limitations

• Synthetic training data is mostly parallel tool combinations, limiting learning of sequential, stateful tool use.
• The 8-turn, ~5000-token context limit is below the API count needed for AppWorld tasks.
• AppWorld gains are modest given stateful APIs not covered by training data.

Experimental setup

Datasets: StableToolBench AppWorld

Benchmarks: StableToolBench AppWorld

Metrics: F1 Recall Precision Match SRecall TGC SGC

Baselines: RAG RAG-SFT Multi-turns Search-R1 GSPO GDPO MARAG-R1

Builds on: GRPO Search-R1 GSPO GDPO MARAG-R1

Closest papers by meaning

• ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.93)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.92)
• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.91)
• AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.91)
• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning (ICLR 2026 · similarity 0.91)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)