Research map / Multi-agent systems and games / LLM agents

Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InteractCS-RL, a user-simulator-based online RL framework with a hybrid-advantage method (CMPO) that combines outcome utility, generative process rewards, and a PID-Lagrangian voucher-cost constraint for customer-service dialogue agents.

Problem

Task-oriented customer service dialogue must balance empathetic, procedure-compliant communication with operational cost (e.g., compensation vouchers), but SFT on static logs imitates costly behavior and success-rate metrics ignore cost.

Approach

Combine a persona-driven user simulator gym with a multi-granularity advantage (session satisfaction + turn-level generative principle scores − a Lagrange cost penalty whose multiplier is set by a PID controller) optimized with a GRPO-style objective.

Key results

• Qwen-2.5-14B-RL reaches 3.05 user satisfaction and 100% finish rate on Hard FDS scene with 27.5% voucher rate, versus GPT-4.1 at 83.8% finish rate and 70.0% voucher rate.
• On tau^2-bench, Qwen2.5-14B InteractCS-RL improves Retail Pass@1 from 43.9% (SFT) to 47.4% and Airline from 20.0% to 28.0%.
• Removing the PID controller leaves voucher rate at 34.5%, violating the 30% budget, while full CMPO settles at 30.8%.

Surprising findings

• A fixed Lagrange penalty over-suppresses compensation (V-Rate 24.6%) and yields the lowest satisfaction, indicating static penalties hurt user experience.
• SFT on Qwen2.5-14B slightly decreases Retail Pass@1 relative to the base model (44.7% to 43.9%).

Limitations

• Primary evaluation uses a proprietary FoodDeliverService environment with LLM-simulated users and LLM-as-judge scoring, limiting reproducibility.
• Evaluation uses only 80 simulated users with three random runs, so differences of a few points may be within noise.
• Reward and judge depend on a Qwen2.5-32B evaluator, and the paper does not validate judge reliability against humans.

Experimental setup

Datasets: pre-collected anonymized food delivery business dialogue logs tau^2-bench

Benchmarks: FoodDeliverService tau^2-bench (Retail, Airline, Telecom)

Metrics: user satisfaction dialogue finish rate communication quality logic quality voucher rate Pass@1 communicate rate DB rate action reward

Baselines: GPT-4.1 DeepSeek-v3.2 LongCat-Flash Qwen3-235B Qwen2.5-7B Qwen2.5-14B Qwen2.5-7B-SFT Qwen2.5-14B-SFT PPO GRPO CAPO

Builds on: GRPO PPO CAPO Lagrange multiplier CMDP generative reward modeling tau^2-bench

Closest papers by meaning

• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.86)
• Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward (NeurIPS 2025 · similarity 0.86)
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.86)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL (ICML 2024 · similarity 0.86)
• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.86)
• Cost-Effective Communication: An Auction-based Method for Language Agent Interaction (AAAI 2026 · similarity 0.85)