Research map / Large language models / Reasoning and chain of thought

THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes THOR, a tool-integrated math reasoning framework combining a multi-agent data pipeline (TIRGen), hierarchical episode- and step-level RL with GRPO, and inference-time self-correction using code execution feedback.

Problem

LLMs struggle with high-precision numerical and symbolic computation in mathematical reasoning; existing tool-integrated reasoning methods face difficulties constructing policy-aligned TIR data (especially for reasoning models), fine-grained optimization under sparse episode-level rewards, and lever

Approach

A generator-refiner pipeline (TIRGen) produces policy-aligned code-augmented reasoning data; RL jointly optimizes episode-level answer correctness and step-level code generation on failed steps via backtracking-based rollouts; at inference, failed code calls trigger suffix regeneration for self-correction.

Key results

• THOR-Thinking-8B reaches 79.8 average across six math benchmarks vs 74.5 for Qwen3-8B, with AIME 2024 at 77.5 vs 64.2.
• THOR-7B (non-reasoning, Qwen2.5-Math-7B base) averages 61.2 vs 40.9 for Qwen2.5-Math-7B-Instruct and 56.2 for TORL-7B.
• THOR-1.5B averages 49.1 across math benchmarks, above Qwen3-1.7B's 36.4 (non-reasoning) and DeepScaleR-1.5B-Preview's 52.5 is exceeded only on reasoning comparison; THOR-Thinking-1.7B reaches 65.3 vs 61.1 for Qwen3-1.7B.

Surprising findings

• Qwen3-8B in thinking mode uses code tools rarely (code ratio 8.5%), and TIRGen-based cold start raises code use to over 90%.
• Self-rewarded selection using execution pass rate alone matches or beats self-consistency and external PRM-based selection at N=8 for the non-reasoning model.

Limitations

• Only code interpreter tools were studied; search engines and symbolic systems were not explored.
• Experiments limited to models up to 8B due to compute; 32B and 72B not tested.
• Evaluation relies on LLM-as-judge (Qwen3-32B) for answer correctness.

Experimental setup

Datasets: DAPO-Math-17k DeepMath-103k DeepScaleR-40k ToRL-28k MATH500 AIME 2024 AIME 2025 AMC 2023 Minerva Math OlympiadBench HumanEval+ MBPP+

Benchmarks: MATH500 AIME 2024 AIME 2025 AMC 2023 Minerva Math OlympiadBench HumanEval+ MBPP+ LiveCodeBench v6

Metrics: Avg@4 accuracy pass@1 pass@16 code ratio token count FPS

Baselines: TORL-7B ARTIST-7B TATA-7B AutoTIR-7B ZTRL-7B START-32B DeepScaleR-1.5B-Preview DeepSeek-R1-Distill-Qwen-7B Qwen3-8B OpenMath-Nemotron-7B

Builds on: GRPO START ToRL Toolformer

Closest papers by meaning

• Towards Effective Code-Integrated Reasoning (AAAI 2026 · similarity 0.89)
• Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.89)
• In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback (AAAI 2026 · similarity 0.89)
• MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.88)
• Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.88)
• ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (ICLR 2024 · similarity 0.88)