Research map / Large language models / Reasoning and chain of thought

ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReTool, a PPO-based RL framework with interleaved code-interpreter execution during rollouts, preceded by a cold-start SFT on code-integrated reasoning traces, reaching 67.0% on AIME2024 with Qwen2.5-32B-Instruct.

Problem

Reasoning LLMs trained with RL struggle on tasks needing precise computation or structured problem-solving, and SFT-based tool use imitates curated data and does not learn when to call tools adaptively.

Approach

Interleave real-time code-sandbox execution into RL rollouts (code between code tags, interpreter output fed back and masked from loss) so the policy learns when and how to invoke a code interpreter from a simple outcome accuracy reward, after a cold-start SFT on code-integrated traces.

Key results

• ReTool (Qwen2.5-32B-Instruct) reaches 67.0% on AIME2024 and 49.3% on AIME2025 in about 400 RL steps, vs. 40.0% / 36.7% for text-based RL.
• ReTool on DeepSeek-R1-Distill-Qwen-32B reaches 72.5% on AIME2024 and 54.3% on AIME2025.
• Response length drops about 40% (10k to 6k tokens) after RL, while code lines rise nearly fivefold.

Surprising findings

• Model spontaneously learns to self-correct non-executable code after interpreter errors, without explicit training data for this behavior.
• Cold-start SFT model alone (40.9% AIME2024) nearly matches the text-based RL baseline (40.0%).

Limitations

• Evaluation limited to AIME2024/2025 math benchmarks with few problems, and baselines' numbers are copied from literature.
• Reward ignores code executability by design, so tool-use behavior is only indirectly shaped.
• Cold-start dataset construction relies on proprietary judges (DeepSeek-R1, Doubao-1.5-pro) and dataset size is not reported.

Experimental setup

Datasets: Open-Thoughts AIME2024 AIME2025

Benchmarks: AIME2024 AIME2025

Metrics: pass@1 (averaged over 32 repeats) average response length code ratio code lines code invocation timing

Baselines: Qwen2.5-Math-72B-Instruct Qwen2.5-Math-72B-Instruct-TIR Sky-T1 DeepSeek-R1-Zero-Qwen-32B QwQ-32B-Preview s1-32B OpenAI o1-preview Text-based RL (Qwen2.5-32B-Instruct)

Builds on: PPO Open-Thoughts DeepSeek-R1 Qwen2.5-32B-Instruct VeRL

Closest papers by meaning

• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.91)
• Towards Effective Code-Integrated Reasoning (AAAI 2026 · similarity 0.91)
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (NeurIPS 2025 · similarity 0.90)
• Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning (ICML 2026 · similarity 0.89)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning (ICLR 2026 · similarity 0.89)