Research map / Large language models / Reasoning and chain of thought

Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ParetoPO, a two-stage multi-objective RL method (hypervolume-guided dynamic scalarization, then Pareto-rank-based GRPO advantages) that trains tool-using LLM agents to balance answer accuracy and tool-call efficiency.

Problem

Tool-integrated LLM agents are usually aligned to maximize task accuracy alone, ignoring tool-use efficiency, and fixed-weight scalarization can miss Pareto-optimal trade-offs.

Approach

Combines hypervolume-driven adaptive reweighting of scalarized rewards (smoothed hypervolume gain) with non-dominated sorting of rollout groups to set GRPO advantages, so that dominance rather than fixed weights drives credit assignment.

Key results

• On MATH500 with Qwen2.5-Math-7B, ParetoPO reaches 84.6 EM with 1.2 tool calls versus 82.2 EM / 2.2 calls for ToRL-GRPO.
• On NQ with Qwen2.5-7B, ParetoPO reaches 50.1 EM with 0.9 tool calls versus 44.4 EM / 1.0 calls for OTC-GRPO.
• Ablation on MATH500 shows removing Pareto ranking drops EM from 80.0 to 77.0 and removing Stage 2 drops it to 76.3.

Surprising findings

• Reducing tool calls appears to improve accuracy too, suggesting many single-objective methods are off the Pareto frontier.

Limitations

• Theoretical guarantees rest on strong assumptions (e.g., Assumption on approximate scalar optimization and dense preference coverage) that are not verified empirically.
• Results reported on a single run per setting with small benchmarks (AIME has 30 problems), limiting statistical power.
• Only two objectives (accuracy and tool efficiency) are studied.

Experimental setup

Datasets: ToRL math training set NQ HotpotQA 2018 Wikipedia dump

Benchmarks: MATH500 AIME2024 AIME2025 OlympiadBench AMC23 NQ HotpotQA

Metrics: exact match tool usage count

Baselines: Qwen2.5-Math-1.5B-Instruct Qwen2.5-Math-1.5B-Instruct-TIR ToRL-GRPO OTC-GRPO MO-GRPO R1-Base SFT RAG Search-R1

Builds on: GRPO ToRL Search-R1 OTC DAPO

Closest papers by meaning

• PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoning (NeurIPS 2026 · similarity 0.88)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.88)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.88)
• ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems (ICML 2026 · similarity 0.86)
• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.86)