Research map / Large language models / Reasoning and chain of thought

Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TIR-Judge, an end-to-end multi-turn RL framework that trains LLM judges to interleave reasoning with Python code execution, with distilled (SFT-initialized) and zero-distillation iterative variants.

Problem

Text-only LLM judges struggle with precise computation and constraint verification (e.g., counting, math), and prior tool-augmented judges rely on prompting or inference-time tool use rather than RL-trained tool use.

Approach

Train judges end-to-end with DAPO-based multi-turn RL over code-interpreter trajectories, using a reward combining correctness, format, and tool-use validity; a Zero variant bootstraps via iterative rejection-sampling SFT and RL without teacher distillation.

Key results

• TIR-Judge-Zero 4B pairwise reaches 76.3 avg on PPE Correctness, versus 60.4 for Qwen3-4B-Instruct pairwise.
• TIR-Judge 8B reaches 96% of Claude-Opus-4 listwise performance on RewardBench 2 (73.4 vs 76.5 average).
• Simply adding code execution to Qwen3 models yields under 1% gain or negative change, while RL-trained tool use yields substantial gains.

Surprising findings

• Adding code-execution tools to base Qwen3 judges without RL gives negligible or negative gains.
• The zero-distillation TIR-Judge-Zero 4B outperforms its distilled counterpart on pairwise PPE (76.3 vs 71.3).

Limitations

• Limited to a single Python code interpreter tool; tool set not expanded.
• Non-verifiable domains such as Safety and Chat show text-only models performing slightly better.
• Gemini-2.5-Flash tool variant can fail to terminate tool calls, and the tool-call turn limit causes some failures.

Experimental setup

Datasets: HelpSteer3 UltraInteract S1 CodeRM Tulu 3 Safe-RLHF MATH DAPO-Math WebInstruct Loong

Benchmarks: PPE Correctness IFBench CodeJudgeBench RewardBench RM-Bench JudgeBench RewardBench 2 AIME 2024 AIME 2025 BigCodeBench IFEval

Metrics: accuracy best-of-N accuracy

Baselines: Qwen3-8B Qwen3-4B-Instruct-2507 Gemini-2.5-Flash Gemini-2.5-Flash-Tool AgentRM RRM RM-R1 Think-RM J1 Deepseek-GRM Armo-RM Skywork-Reward-Gemma-2

Builds on: DAPO GRPO tool-integrated reasoning Verl-Tool RM-R1

Closest papers by meaning

• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (ICLR 2026 · similarity 0.90)
• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.90)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.89)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)
• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning (NeurIPS 2026 · similarity 0.89)
• A$^2$FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning (ICLR 2026 · similarity 0.89)