Research map / Large language models / Language, safety and interpretability

Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines three LLM-as-a-judge consistency metrics for simulated users and fine-tunes Llama-3.1-8B-Instruct with multi-turn PPO using prompt-to-line consistency as reward, reducing persona inconsistency across open-ended chat, education, and mental health tasks.

Problem

LLM-simulated human users drift from assigned personas, contradict earlier statements, and lose role-appropriate behavior, which undermines their use in training and evaluating downstream agents.

Approach

Treats the user simulator as a trainable agent rather than a fixed environment, defines three judge-based consistency metrics (prompt-to-line, line-to-line, Q&A), and uses them as turn-level rewards for multi-turn PPO fine-tuning of the simulated user.

Key results

• PPO improves prompt-to-line consistency over the Llama-3.1-8B-Instruct baseline by +58.5% on open-ended conversation, +20.6% on education, and +37.6% on mental health.
• LLM judge achieves 80.03% average agreement with humans, with average Fleiss' kappa 0.552 across consistency definitions.
• Abstract claims over 55% reduction in inconsistency after multi-turn RL fine-tuning.

Measured effects reported in the paper

Surprising findings

• Human-human agreement (average Fleiss' kappa 0.303) is lower than model-human agreement, so the LLM judge was more reliable than individual human raters in this study.

Limitations

• Optimizes a static persona notion and may penalize justified persona changes in realistic dialogue.
• Human evaluation relied on a limited pool of about thirty annotators, which may under-represent judgment diversity.
• Fine-tuning only targets prompt-to-line consistency; joint optimization of all metrics is left for future work.

Experimental setup

Datasets: PersonaChat-style personas synthetic student personas (27 learning styles) synthetic patient conditions (100) self-generated dialogues

Benchmarks: prompt-to-line consistency on open-ended conversation education student simulation mental health patient simulation

Metrics: prompt-to-line consistency line-to-line consistency Q&A consistency percent agreement Fleiss' kappa

Baselines: Llama-3.1-8B-Instruct SFT KTO

Builds on: PPO KTO OpenRLHF PersonaChat

Closest papers by meaning

• Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment (NeurIPS 2025 · similarity 0.89)
• Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts (NeurIPS 2026 · similarity 0.88)
• Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward (NeurIPS 2025 · similarity 0.88)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.87)
• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents (ICLR 2024 · similarity 0.87)
• OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.87)