Research map / Large language models / Alignment and preferences

Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLPA, a PPO-based multi-turn RL framework where a 3B model infers and refines slot-value user profiles through dialogue with a simulated user, trained with profile and response rewards.

Problem

Personalized alignment of LLMs in cold-start settings and over long-term dialogue, where prompt-based and offline fine-tuning methods are static and shallow.

Approach

Formulates personalized alignment as a multi-turn MDP and trains the LLM online with a simulated user, using a dual reward per turn: a slot-wise F1 profile reward for inferred user profiles and a multiplicative binary-criteria response reward judged by an LLM.

Key results

• On Vanilla ALOE, Qwen-RLPA (3B) reaches an average alignment score of 73.38, versus 45.27 for DPO and 44.32 for SFT on the same backbone, and 75.81 for GPT-4o-mini with Self-Critic.
• On Extended ALOE, RLPA scores 52.74 average and leads all three metrics, versus 27.26 for DPO, while GPT-4o reaches 54.66 average with lower N-R^2.
• Removing the Response Reward drops average alignment to 66.19 and removing the Profile Reward to 45.54, versus 73.38 for the full RLPA.

Measured effects reported in the paper

Surprising findings

• GPT-4o and GPT-4o-mini achieve slightly higher raw alignment than RLPA in the Vanilla setting, though RLPA has higher N-R^2.
• Ablation: Response Reward alone yields the best-in-ablation early-turn performance but the Profile Reward alone performs poorly (45.54 average).

Limitations

• Assumes a single-user interaction thread; multi-user, multi-session and cross-domain personalization not studied.
• Theoretical understanding of long-term convergence is not developed.
• Relies on GPT-4o-mini as both user simulator and reward model, with evaluation judged by GPT-4o.

Experimental setup

Datasets: ALOE training set ALOE test set PersonaChat metadata

Benchmarks: ALOE Vanilla Extended ALOE

Metrics: alignment score (AVG.) N-IR N-R^2 profile score

Baselines: Reminder Self-Critic Chain-of-Thought RAG SFT DPO GPT-4o Claude-3.5-Sonnet DeepSeek-R1 QwQ-32B

Builds on: PPO ALOE PrefEval OpenRLHF

Closest papers by meaning

• Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.90)
• MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.89)
• Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward (NeurIPS 2025 · similarity 0.89)
• Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.89)