Research map / Multi-agent systems and games / LLM agents

Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Behavioral Agentic Optimization (BAO), combining behavior-enhanced SFT (retrospective reasoning and prospective planning traces from GPT-4o) with GRPO using behavior regularization penalties, to push the Pareto frontier between task performance and user engagement on UserRL proactive tasks.

Problem

Proactive LLM agents trained with agentic RL face a trade-off between task performance and user engagement, since frequent user verification improves accuracy but burdens users, and naive user-engagement penalties hurt exploration.

Approach

Instead of handcrafted engagement reward weights, enhance inter-turn behaviors (memory management, hypothesis refinement, dynamic scheduling, strategic querying) via SFT from a teacher model, then regularize them during GRPO with turn-level penalties for consecutive user requests without environment info gain and for premature termination/over-thinking.

Key results

• On Function-Gym, BAO-4B reaches Pass@U-1 0.2692 and score 0.6923 with UR 0.2148, versus UserRL-4B at 0.2436, 0.5256 and UR 0.3758.
• On Turtle-Gym, BAO-4B lowers UR to 0.2917 from UserRL-4B's 0.7617 while raising Pass@U-1 from 0.0417 to 0.1063.
• With GPT-4o-mini as teacher on Telepathy-Gym (Qwen3-1.7B), BAO raises Pass@U-1 from 0.1707 (UserRL) to 0.5122 and lowers UR from 0.7619 to 0.1423.

Surprising findings

• Removing behavior regularization leads to a large increase in user involvement rate (e.g., Turtle-Gym UR 0.8698 for BAO w/o BR-4B), indicating reliance on user verification.
• Naive penalties on user interactions were reported to harm task performance by limiting exploration.

Limitations

• Focuses only on language-only agents; multimodal extension left for future work.
• Reward hacking risk when using Qwen3-8B as judge and simulator during training, with sim-to-real gap to GPT-4o evaluation.
• Evaluated only on three UserRL gym tasks.

Experimental setup

Datasets: UserRL benchmark (Function-Gym, Telepathy-Gym, Turtle-Gym)

Benchmarks: UserRL Function-Gym UserRL Telepathy-Gym UserRL Turtle-Gym

Metrics: Pass@U-1 Pass@U-2 score user involvement rate Self-BLEU embedding similarity reward translation rate

Baselines: UserRL Gemini-3-flash Gemini-2.5-Pro Gemini-2.5-Flash GPT-5-0807 GPT-4o GPT-4o-mini Qwen3-32B Qwen3-14B Qwen3-8B

Builds on: UserRL GRPO CollabLLM

Closest papers by meaning

• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.91)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.89)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.89)
• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents (ICLR 2024 · similarity 0.88)