Research map / Reinforcement learning / RL from feedback

PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PORTool, which trains multi-tool agents with GRPO-style RL over a bounded-branching rewarded rollout tree, adding fork-relative advantages from prefix-consistent step importance to trajectory-relative advantages.

Problem

Outcome-only trajectory rewards in agentic RL for multi-tool-integrated reasoning cause credit-assignment ambiguity, broadcasting one reward to all tool-call steps and possibly unlearning necessary steps.

Approach

Organize rollouts into a rewarded prefix-sharing tree so sibling tool-call candidates at each fork are compared under identical histories; assign each step an importance score aggregated from descendant outcomes plus bounded format/execution reward (adaptive max/avg aggregation), and combine fork-relative with trajectory-relative advantages.

Key results

• On Qwen-2.5-7B-Instruct, PORTool reaches 51.98% accuracy vs 48.23% for the best baseline (AgentFlow) with 3.07 tool-call steps and 7.10% unanswerable rate.
• On Qwen-3-1.7B, PORTool reaches 46.60% accuracy, improving over GRPO_fm (42.76%) by 3.84 points, with 2.64 tool-call steps.
• On ToolBench average pass rate, PORTool achieves 52.8% vs 50.9% for the strongest baseline (ARPO_fm).

Surprising findings

• Formatting-augmented ARPO (ARPO_fm) underperforms plain ARPO on the 7B model in accuracy (44.97 vs 48.18).

Limitations

• Evaluation relies on GPT-4o as judge (94.37% agreement with humans on 700 pairs) on a proprietary in-house dataset.
• Dataset is proprietary and not released, limiting reproducibility.
• Requires tree rollouts and sibling-based normalization; cost of branching not reported.

Experimental setup

Datasets: in-the-wild virtual assistant queries (3,072 train / 535 eval) ToolBench TravelPlanner

Benchmarks: in-house virtual assistant eval (535 queries) ToolBench TravelPlanner

Metrics: final-answer accuracy average tool-call steps unanswerable rate formatting reward pass rate delivery rate

Baselines: GRPO DAPO ARPO AgentFlow SimpleTIR ToRL Prompting

Builds on: GRPO ARPO ToolRL GiGPO Search-R1

Closest papers by meaning

• A$^2$FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning (ICLR 2026 · similarity 0.89)
• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.89)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.89)
• Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization (ICML 2026 · similarity 0.88)
• ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning (ICLR 2026 · similarity 0.88)