Research map / Multi-agent systems and games / LLM agents

Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward

NeurIPS 2025 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CURIO, which adds a curiosity-based intrinsic reward (potential-based reward shaping over a user model's belief improvement) to multi-turn RLHF so LLM agents learn about users during conversation, evaluated on Exercise Recommendation and Education Dialogue.

Problem

Standard RLHF optimizes a single user-agnostic reward and sparse end-of-conversation signals, so LLM agents fail to personalize online for users whose traits are unknown in advance and generalize poorly to new users.

Approach

Use a parameterized user model to estimate belief over user types after each turn and reward the policy with potential-based shaping on belief improvement (e.g., log-accuracy difference), giving dense intrinsic rewards that encourage asking informative questions without changing the optimal policy.

Key results

• On Exercise Recommendation, DiffAcc reaches 87.5% success vs 54.0% SFT and 68.5% MTRLHF.
• On Education Dialogue personalization (Auto Eval), DiffLogAcc beats MTRLHF with 75.95% win rate; human study gives 76% (152/200, p<.001).
• DiffLogAcc is preferred over the MTRLHF baseline on conversation quality in auto eval (54.43% vs 45.57%), while human quality judgments showed no significant difference (45%, p=.179).

Measured effects reported in the paper

Surprising findings

• Non-potential-based entropy and accuracy shaping led the policy to lengthen conversations to farm intrinsic reward.
• Ungrounded entropy-based rewards produced controlling behavior where the teacher forced one learning style regardless of the student's true type.

Limitations

• Assumes pre-defined, static user traits.
• Relies on LLM-based user simulators for training and testing, so synthetic rewards may diverge from human preferences.
• Ungrounded personalization can produce controlling behavior and reward hacking.

Experimental setup

Datasets: Education Dialog dataset (prior work) Exercise Recommendation simulated users (generated)

Benchmarks: Exercise Recommendation Education Dialogue

Metrics: success rate pairwise win rate validation accuracy generalization gap user type prediction accuracy

Baselines: SFT MTRLHF InfoGain Entropy reward shaping Acc reward shaping

Builds on: multi-turn RLHF potential-based reward shaping VIME generalized advantage estimation REINFORCE

Closest papers by meaning

• Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment (NeurIPS 2025 · similarity 0.89)
• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.89)
• Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
• DialogXpert: Driving Intelligent and Emotion-Aware Conversations Through Online Value-Based Reinforcement Learning with LLM Priors (AAAI 2026 · similarity 0.87)
• Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.87)