Research map / Reinforcement learning / RL from feedback

Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes G2PO, a group-based step-level agentic RL method that merges identical observations across sampled trajectories into a state-transition graph, averages state values over each group, and globally standardizes edge TD errors for advantage estimation.

Problem

Long-horizon agentic RL suffers from sparse, delayed rewards; existing step-level group methods (e.g., GiGPO) estimate state values from single trajectories and compare actions only locally, causing high-variance value estimates and myopic credit assignment.

Approach

Build a global state-transition graph by clustering identical observations across trajectories; estimate each state's value as the mean of its group's discounted outcomes, treat actions as edges, and standardize edge TD errors over the whole graph (edge-centric advantage) combined with node-centric and episode-level advantages.

Key results

• On WebShop with Qwen2.5-1.5B, G2PO reaches 71.2% success vs 56.8% for GRPO (+14.4); on ALFWorld overall success improves by +22.2 points over GRPO at 1.5B.
• On ALFWorld with Qwen2.5-7B, G2PO achieves 96.9% overall success vs 77.6% for GRPO and 90.8% for GiGPO.
• On AppWorld with Qwen2.5-14B, G2PO achieves 27.6% success vs 24.8% for GRPO and 25.7% for GiGPO.

Surprising findings

• Frontier models such as Qwen3.5-397B (17.2%) and Gemini-2.5-Pro (35.9%) score far below small RL-trained agents on WebShop success.

Limitations

• Implemented only on the verl-agent framework.
• Not evaluated on tree-based or asynchronous rollout settings.
• Relies on identical observations to form state groups, so benefit depends on observation repetition.

Experimental setup

Datasets: WebShop ALFWorld AppWorld

Benchmarks: WebShop ALFWorld AppWorld

Metrics: success rate score number of interaction turns

Baselines: Qwen3.5-397B DeepSeek-V3.2 Gemini-2.5-Pro ReAct Reflexion PPO RLOO GRPO GiGPO

Builds on: GRPO GiGPO RLOO PPO verl-agent

Closest papers by meaning

• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.93)
• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.89)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.89)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.89)
• TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios (ICML 2026 · similarity 0.88)
• StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction (NeurIPS 2026 · similarity 0.88)