Research map / Reinforcement learning / Visual and representation RL

Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues

AAAI 2025 · Reinforcement learning › Visual and representation RL · approach: Inferring decision-making from behavior · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LDPP, a simulation-free framework that discovers fine-grained latent dialogue policies from raw dialogue records with a VQ-VAE-style encoder and plans them via offline hierarchical RL with a P-Former adapter on a frozen LLM.

Problem

Proactive dialogue policy planning (emotional support, persuasion) relies on LLM user simulators and predefined coarse-grained policies, which are costly, biased, and inefficient to train.

Approach

Automatically discover continuous latent policies from real dialogue records (weighted sum over a codebook), annotate the data with them, and train a policy planner and generator offline with a hierarchical IQL/advantage-weighted objective, bridging latent policies to a frozen LLM via a trainable P-Former.

Key results

• On ExTES, LDPP (Qwen1.5-1.8B) reaches SSR 0.723, SR 0.903, AvgT 4.132, beating ChatGPT (SR 0.810) and PPDPP (SR 0.558).
• On P4G, LDPP achieves SR 0.795 and SSR 0.733 versus PPDPP SR 0.745 and ChatGPT SR 0.460 (gpt-3.5-turbo-0613 critic).
• Removing the 3rd (offline hierarchical RL) stage drops ExTES SR from 0.903 to 0.623, showing it is the key component.

Measured effects reported in the paper

Surprising findings

• LDPP without the 2nd (policy distillation) stage performed better than full LDPP under the gpt-3.5-turbo-0125 critic, suggesting distillation can be detrimental in some settings.
• Increasing the number of policy tokens T generally decreased success rate, indicating policy tokens add noise to the LLM.

Limitations

• Evaluation relies on ChatGPT critic and self-play simulation, which may still carry LLM bias.
• P4G training set is expanded with ChatGPT-generated synthetic dialogues.
• Latent policies lack explainability; human evaluation notes the planner rarely asks specific questions and gives suggestions too quickly.

Experimental setup

Datasets: ExTES P4G ESConv

Benchmarks: ExTES ESConv P4G

Metrics: SSR SR AvgT human win/lose rate

Baselines: Proactive ProCoT PPDPP Standard Prompt ICL-AIF LoRA finetuning ChatGPT

Builds on: VQ-VAE IQL PPDPP REINFORCE

Closest papers by meaning

• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents (ICLR 2024 · similarity 0.92)
• Multimodal Policy Internalization for Conversational Agents (ICLR 2026 · similarity 0.89)
• DialogXpert: Driving Intelligent and Emotion-Aware Conversations Through Online Value-Based Reinforcement Learning with LLM Priors (AAAI 2026 · similarity 0.88)
• IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learning (ICML 2026 · similarity 0.87)
• Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents (ICLR 2026 · similarity 0.87)
• Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF (ICLR 2025 · similarity 0.87)