Research map / Reinforcement learning / RL from feedback

Group-in-Group Policy Optimization for LLM Agent Training

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GiGPO, a critic-free group-based RL algorithm for multi-turn LLM agents that adds step-level advantages via anchor state grouping of repeated environment states, achieving gains over GRPO on ALFWorld, WebShop, and search QA.

Problem

Group-based RL methods such as GRPO assign credit only at the episode level, which is inadequate for long-horizon, sparse-reward multi-turn LLM agent training.

Approach

Two-level group-in-group advantage: episode-level relative advantages over full trajectories plus step-level advantages computed over actions taken from identical environment states (anchor states) found retroactively across trajectories via hashmap grouping, requiring no extra rollouts or critic.

Key results

• GiGPO w/o std improves over GRPO by 13.3% on ALFWorld and 10.6% on WebShop success at Qwen2.5-1.5B (86.1% vs 72.8% ALFWorld overall).
• GiGPO reaches 42.1% average on search QA with Qwen2.5-3B and 47.2% with Qwen2.5-7B, above Search-R1 (32.5%, 38.5%).
• Step-level grouping overhead is 0.01s hashmap plus 0.53s advantage computation per iteration, under 0.002% of the ~363s iteration time.

Measured effects reported in the paper

Surprising findings

• Fixing the normalization factor to 1 (leave-one-out, RLOO-like) outperforms std normalization on harder tasks such as Look, Pick2, and WebShop, while std is no better elsewhere.
• Over 65% of environment states recur across trajectories in the group, so step-level groups are commonly non-trivial.

Limitations

• Relies on exact state matching for anchor grouping, which may fail in noisy or highly complex environments; similarity-based grouping is only a partial fix.
• Degrades to GRPO when no states repeat across trajectories.
• Best omega and normalization choice appear task-dependent.

Experimental setup

Datasets: ALFWorld WebShop NQ HotpotQA TriviaQA PopQA 2Wiki MuSiQue Bamboogle Sokoban EZPoints

Benchmarks: ALFWorld WebShop NQ TriviaQA PopQA HotpotQA 2Wiki MuSiQue Bamboogle Sokoban EZPoints

Metrics: success rate score exact match accuracy training time per iteration

Baselines: GPT-4o Gemini-2.5-Pro ReAct Reflexion PPO RLOO GRPO Search-R1 ZeroSearch StepSearch R1-Instruct DAPO

Builds on: GRPO RLOO PPO veRL DAPO

Closest papers by meaning

• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning (NeurIPS 2026 · similarity 0.93)
• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.93)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.91)
• Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.91)
• Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (NeurIPS 2026 · similarity 0.91)
• Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents (ICLR 2026 · similarity 0.90)