Research map / Large language models / Reasoning and chain of thought

SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SimpleTIR, a plug-and-play trajectory filter that drops whole multi-turn tool-integrated reasoning trajectories containing a "void turn" (no code block and no final answer) from the GRPO/PPO update, stabilizing Zero-RL training and reaching AIME24 50.5 from Qwen2.5-7B.

Problem

Multi-turn tool-integrated reasoning (TIR) trained with end-to-end RL from base models is unstable, with gradient norm explosions and performance collapse, attributed to out-of-distribution tool feedback causing low-probability tokens that compound over turns.

Approach

Identify "void turns" (LLM responses with neither a complete code block nor a final answer) as a signature of distributional drift, and mask the policy loss for the entire trajectory containing any void turn. This blocks high-magnitude gradients from low-probability tokens and avoids penalizing earlier correct turns for later collapse.

Key results

• SimpleTIR-7B reaches 50.5 on AIME24 starting from Qwen2.5-7B, up from 22.1 for the text-only baseline and above ZeroTIR-7B (39.6).
• SimpleTIR-32B reaches 59.9 on AIME24 and 49.2 on AIME25, with 92.9 on MATH500.
• Ablation on AIME24: naive multi-turn 20.8, low-prob filtering 23.3, high-ratio filtering 26.3, stop-generation without filtering 26.1, versus 50.5 for full SimpleTIR-7B.

Surprising findings

• Filtering by low-probability tokens or high importance ratio does not resolve multi-turn TIR instability, despite being effective in other settings.
• Naive multi-turn Zero-RL TIR underperforms the single-turn text-only baseline on AIME24 (20.8 vs. 22.1 text-only Qwen2.5-7B baseline reported in abstract).

Limitations

• Void-turn indicator may not transfer to tasks beyond multi-turn TIR.
• Maximum turns restricted to 10 for math; more interactions may be needed for complex agent tasks.
• Depends on a highly parallel sandbox for code execution; rollout asynchrony remains open.

Experimental setup

Datasets: SimpleRL math data Deepscaler

Benchmarks: AIME24 AIME25 MATH500 AMC23 HMMT Feb 25 Olympiad

Metrics: average@32 accuracy

Baselines: DAPO SimpleRL-Zoo ToRL Effective TIR ARPO ZeroTIR ReTool Qwen2.5-7B-TIR Naive Multi-Turn

Builds on: GRPO PPO DAPO VeRL Search-R1

Closest papers by meaning

• Towards Effective Code-Integrated Reasoning (AAAI 2026 · similarity 0.90)
• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning (ICLR 2026 · similarity 0.90)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning (ICLR 2026 · similarity 0.89)
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.89)
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (ICLR 2026 · similarity 0.88)