Research map / Multi-agent systems and games / LLM agents

InfoPO: Information-Driven Policy Optimization for User-Centric Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InfoPO, a multi-turn RL method that adds a turn-level counterfactual information-gain reward (masked-feedback log-prob shift) fused with outcome advantages via a variance-gated weight, improving user-centric agents on UserGym, ColBench, and tau2-Bench.

Problem

Multi-turn GRPO for user-centric agents with underspecified requests suffers from trajectory-level rewards, causing credit assignment problems and near-zero advantages when rollout groups have uniform outcomes.

Approach

Scores each turn by how much the user/environment feedback shifts the policy's log-probability of the realized next action versus a masked-feedback counterfactual (teacher forcing, no extra rollouts), and adaptively weights this info-gain advantage with a sigmoid gate on the within-group std of outcome rewards.

Key results

• InfoPO achieves average 0.455 on Qwen2.5-7B-Instruct versus 0.410 for RAGEN and 0.397 for Search-R1 across UserGym, ColBench, and tau2-Bench.
• On Qwen3-4B, InfoPO averages 0.481 versus 0.416 for UserRL.
• Ablation: removing extrinsic reward (w/o R_ext) drops Qwen2.5-7B average to 0.312, below all RL baselines.

Surprising findings

• Pure information-gain reward without extrinsic outcome reward (w/o R_ext) performs worse than prompting and most RL baselines, showing information seeking alone is insufficient.
• Removing the variance gate leads to more generic open-ended questions and longer interactions (reward-hacking pattern).

Limitations

• Assumes behavior policy and user/environment have non-zero support over task-relevant continuations; uninformative rollouts yield no intrinsic signal.
• Evaluation relies on simulated users (GPT-4o-mini) and simulated benchmarks; experiments only in simulation.
• Abstract claims 14-16% gain over GRPO methods while tables show smaller margins on some tasks; stronger user simulators make UserGym tasks mixed.

Experimental setup

Datasets: UserGym ColBench tau2-Bench Sokoban WebShop

Benchmarks: UserGym ColBench tau2-Bench Sokoban WebShop

Metrics: success rate accumulated reward Pass Succ. Avg@4

Baselines: GPT-4.1 Gemini-3-Flash GPT-4o-mini ReAct Reflexion RAGEN Search-R1 UserRL

Builds on: GRPO UserRL IGPO RAGEN

Closest papers by meaning

• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.88)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.88)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.87)
• MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems (ICML 2026 · similarity 0.87)
• TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.87)
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents (ICML 2026 · similarity 0.87)