Research map / Large language models / Reasoning and chain of thought

AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AutoTool, an SFT warm-up followed by GRPO-style RL with decoupled adaptive entropy constraints that lets a Qwen2.5-7B tool-use model automatically choose between think and no-think modes, improving BFCL accuracy while cutting tokens.

Problem

Direct RL for tool use suffers reasoning collapse (shrinking thinking length), while scaled-up distilled reasoning models overthink simple tool-use queries at high token cost.

Approach

Warm-up SFT on mixed think/no-think data, then RL with an entropy loss decoupled by trajectory type: fixed entropy coefficient for short responses and an adaptively updated coefficient for long reasoning to keep entropy above a target, plus an asymmetric mode-dependent answer reward.

Key results

• AutoTool-7B reaches 70.12 overall BFCL accuracy versus 53.69 for Qwen2.5-7B-Instruct and 58.17 for PubTool-SFT.
• AutoTool uses about 183 tokens versus about 966 for the distilled model, a roughly 81% token reduction.
• Thinking rate is 45% on Multi-Turn and 0% on Non-Live, indicating difficulty-dependent mode selection.

Surprising findings

• Direct GRPO on tool use shows response length collapsing while accuracy rises, contrary to math RL trends.
• A fixed length-penalty constraint did not improve performance, while the entropy constraint did.

Limitations

• Evaluated only on a 7B model size in the main experiments; scalability to other sizes and architectures untested.
• Generalization beyond tool-use tasks (e.g., math) not demonstrated.
• Only tested with GRPO; compatibility with other RL algorithms such as PPO or DAPO untested.

Experimental setup

Datasets: ToolACE xLAM APIGen-MT-5k Hermes Function-Calling PubTool

Benchmarks: BFCL API-Bank ACEBench

Metrics: accuracy overall accuracy thinking rate token cost ACU

Baselines: Qwen2.5-7B-Instruct PubTool-SFT PubTool-Distilled PubTool-GRPO Thinkless AdaCtrl ToolRL-7B Tool-N1-7B Qwen3-8B GPT-4o o3 Gemini-2.5-Pro

Builds on: GRPO ToolACE Thinkless AdaCtrl

Closest papers by meaning

• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.89)
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (ICLR 2026 · similarity 0.88)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.88)
• Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL (NeurIPS 2025 · similarity 0.88)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.87)
• AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.87)