Research map / Reinforcement learning / RL from feedback

Universal Post-Processing Networks for Joint Optimization of Modules in Task-Oriented Dialogue Systems

AAAI 2025 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniPPN, a single GPT-2-based post-processing network that modifies outputs of all modules in task-oriented dialogue systems, trained with imitation learning on pseudo-demonstrations and module-level PPO, outperforming BinPPN+GenPPN on MultiWOZ simulation and human evaluation.

Problem

Existing post-processing networks (PPNs) for task-oriented dialogue systems only cover subsets of modules (BinPPN for NLU/DST/policy, GenPPN for NLG) and cannot be jointly optimized, and GenPPN needs dense DA-level feedback that end-to-end systems do not expose.

Approach

A single language-model-based network post-processes outputs of every module as a sequence-transformation task, trained by imitation learning on pseudo-demonstrations (negatives sampled from similar turns, plus copy tokens) and then PPO under a module-level MDP with per-module value and advantage estimates from sparse task-success reward.

Key results

• On SYS_RULE, UniPPN raised success rate from 83.69 to 90.62 versus 85.25 for BinPPN&GenPPN.
• On SYS_D3ST, UniPPN raised success from 58.50 to 85.06 (BinPPN&GenPPN: 67.58).
• In human evaluation with SYS_PPO, UniPPN reached 54.17% success and 8.06 turns (significant at p<0.1) versus 46.51% and 8.79 for the original system.

Measured effects reported in the paper

Surprising findings

• BinPPN&GenPPN performed worse than the original SYS_PPO on most human-evaluation metrics, suggesting disjoint PPN training hurts coordination.
• GenPPN achieved fewer turns than UniPPN on NLG post-processing, though success rates were similar.

Limitations

• Reward only reflects task completion, so subjective satisfaction (LU, RA, OS) did not improve over the original system.
• Evaluated only on MultiWOZ with text dialogue and a user simulator; human evaluation limited to one system (SYS_PPO).
• Training cost still high (many simulated dialogues needed for convergence).

Experimental setup

Datasets: MultiWOZ

Benchmarks: MultiWOZ simulation with ConvLab-2 agenda-based user simulator

Metrics: success rate inform recall inform precision inform F1 book rate match rate turns human LU/RA/OS Likert

Baselines: BinPPN&GenPPN BinPPN GenPPN original dialogue systems without post-processing

Builds on: BinPPN GenPPN PPO generalized advantage estimation MultiWOZ

Closest papers by meaning

• Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning (AAAI 2024 · similarity 0.84)
• Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.83)
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.83)
• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.83)
• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents (ICLR 2024 · similarity 0.83)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.83)