Research map / Large language models / Agents, code and math
Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PPDPP, a tunable RoBERTa-based dialogue policy planner plug-in trained with SFT and RL from goal-oriented AI feedback via LLM self-play, and an interactive evaluation protocol for proactive dialogues.
Problem
Improving dialogue policy planning (proactivity) of LLM-powered dialogue agents in goal-oriented proactive dialogues such as negotiation, emotional support, and tutoring, where prompt-based methods are capped by frozen LLMs and do not transfer across cases.
Approach
A small learnable plug-in policy planner (RoBERTa) predicts the next dialogue strategy for a frozen LLM agent; it is trained by SFT on human dialogues and then by policy gradient using rewards from an LLM reward model judging goal completion during LLM-vs-LLM self-play, so the learned plug-in transfers across cases and applications without updating the LLM.
Key results
- On CraigslistBargain, PPDPP achieves SL% 0.3376 and SR 0.6117 versus 0.1588 SL% and 0.3830 SR for Standard prompting with ChatGPT.
- On ESConv, PPDPP reaches SR 0.8462 with AT 4.56, compared with 0.7692 SR for Standard prompting.
- On CIMA, PPDPP reaches SR 0.8407 with AT 3.03, compared with 0.6903 SR for Standard prompting; RL is the main source of gains over SFT-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reinforcement learning with simulated interactions (ablation) | PPDPP · CraisglistBargain | SL%: 0.2280 -> 0.3376 | Helped |
| add reinforcement learning with simulated interactions (ablation) | PPDPP · CraisglistBargain | SR: 0.6649 -> 0.6117 | Hurt |
| add reinforcement learning with simulated interactions (ablation) | PPDPP · ESConv | SR: 0.7308 -> 0.8462 | Helped |
| add reinforcement learning with simulated interactions (ablation) | PPDPP · CIMA | SR: 0.7965 -> 0.8407 | Helped |
| add supervised fine-tuning on human-annotated dialogues (ablation) | PPDPP · CraisglistBargain | SR: 0.6223 -> 0.6117 | Hurt |
| add supervised fine-tuning on human-annotated dialogues (ablation) | PPDPP · ESConv | SR: 0.8384 -> 0.8462 | Helped |
| add supervised fine-tuning on human-annotated dialogues (ablation) | PPDPP · CIMA | SR: 0.8230 -> 0.8407 | Helped |
| sample reward feedback l=10 times rather than once (ablation) | PPDPP · CraisglistBargain | SL%: 0.2623 -> 0.3376 | Helped |
| sample reward feedback l=10 times rather than once (ablation) | PPDPP · ESConv | SR: 0.8307 -> 0.8462 | Helped |
| sample reward feedback l=10 times rather than once (ablation) | PPDPP · CIMA | SR: 0.7965 -> 0.8407 | Helped |
| use goal-oriented AI feedback with dynamic multi-turn interaction instead of fixed prompting | PPDPP · ESConv | SR: 0.7308 -> 0.8462 | Helped |
| use goal-oriented AI feedback with dynamic multi-turn interaction instead of fixed prompting | PPDPP · CIMA | SR: 0.7965 -> 0.8407 | Helped |
Surprising findings
- SFT on human-annotated corpora alone performs worse than Standard prompting on ESConv (SR 0.7308 vs 0.7692).
- ICL-AIF gives only trivial deal-benefit gains and even lowers deal success rate on negotiation.
Limitations
- Evaluation relies on ChatGPT as the reward model and user simulator, whose reliability is validated only on small samples (50 dialogues per dataset).
- Vicuna and LLaMA2 fail to judge Italian translation correctness, limiting open-source reward modeling for tutoring.
- RL optimizing SL% lowers deal success rate in negotiation, showing a trade-off between objectives.
Experimental setup
Datasets: CraigslistBargain ESConv CIMA
Benchmarks: CraigslistBargain ESConv CIMA
Metrics: average turn (AT) success rate (SR@8) sale-to-list ratio (SL%) human win rate
Baselines: Standard prompting DialoGPT Proactive ProCoT ProCoT + MI-Prompt Ask-an-Expert ICL-AIF
Builds on: ProCoT Ask-an-Expert ICL-AIF RLAIF
Closest papers by meaning
- Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues (AAAI 2025 · similarity 0.92)
- DialogXpert: Driving Intelligent and Emotion-Aware Conversations Through Online Value-Based Reinforcement Learning with LLM Priors (AAAI 2026 · similarity 0.89)
- Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.88)
- Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.87)
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.