Research map / Large language models / Reasoning and chain of thought

Agentic Reinforced Policy Optimization

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ARPO, an agentic RL algorithm that branches extra rollouts at high-entropy tool-call steps and assigns shared/individual advantages, improving multi-turn tool-use agents while halving tool calls.

Problem

Trajectory-level RL (GRPO, DAPO, REINFORCE++) for multi-turn tool-using LLM agents neglects step-level exploration after tool-call feedback, which introduces high token entropy and uncertainty.

Approach

Entropy-based adaptive rollout: sample globally, monitor token-entropy change after each tool call, and branch partial rollouts at steps whose entropy rises above a threshold; combine with soft/hard advantage attribution so shared prefix tokens and branched tokens get distinct credit.

Key results

• On Qwen2.5-7B-Instruct, ARPO averages 58.3 across 10 reasoning tasks vs. 56.5 for GRPO and 54.8 for DAPO.
• With Qwen3-14B on deep search, ARPO reaches 43.7 average on GAIA-style tasks vs. 36.9 for GRPO (per table average), and 56.4 on GAIA Lv.1.
• ARPO matches or exceeds GRPO accuracy on Qwen2.5-7B using only about half the tool calls.

Surprising findings

• Prompt-based tool-integrated reasoning (TIR) often underperforms direct reasoning for Qwen and Llama models.
• Search feedback induces higher entropy uncertainty than Python feedback.

Limitations

• Evaluation of deep search relies on LLM-as-judge (Qwen2.5-72B) and small test splits.
• Hyperparameters (entropy weight, threshold, alpha) tuned in limited scaling study on Qwen2.5-7B only.

Experimental setup

Datasets: Tool-Star SFT dataset (54K) STILL (0.8K) Tool-Star RL dataset (10K) SimpleDeepSearcher WebSailor

Benchmarks: AIME24 AIME25 MATH500 GSM8K MATH WebWalker HotpotQA 2WikiMultihopQA MuSiQue Bamboogle GAIA HLE

Metrics: accuracy F1 pass@1 pass@3 pass@5 tool-call count

Baselines: GRPO DAPO REINFORCE++ TIR prompting RAG Search-o1 WebThinker ReAct QwQ-32B DeepSeek-R1-671B GPT-4o o1-preview

Builds on: GRPO DAPO REINFORCE++ Tool-Star VERL

Closest papers by meaning

• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.91)
• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.90)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.90)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.90)
• ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
• Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization (ICLR 2026 · similarity 0.90)