Research map / Large language models / Alignment and preferences
PersonalLLM: Tailoring LLMs to Individual Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PersonalLLM, a benchmark of 10,402 open-ended prompts with 8 high-quality LLM responses each, and simulates diverse users as Dirichlet-weighted ensembles of 10 reward models, with ICL and meta-learning baselines.
Problem
Existing alignment benchmarks assume uniform human preferences, so there is no testbed for personalizing LLM responses to heterogeneous individual users with sparse feedback.
Approach
Generate many high-quality responses per prompt from top LLMs so humans are expected to disagree, and simulate a large user base by sampling Dirichlet weightings over 10 pre-trained reward models, yielding more diverse preferences than persona prompting.
Key results
- Simulated users (alpha=0.05) yield a majority-vote top response for only about half of prompts, versus about 90% for persona prompting.
- Simulated user population achieves a representativeness score of 0.839 on OpinionQA overall US population, higher than any LLM in the original study.
- In personalized ICL, winning-only examples performed best; in meta-learning, the strongest user embedding method beats self-ICL, though gains are small.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use weighted ensembles of reward models to simulate users instead of persona prompting | PersonalLLM preference simulation · PersonalLLM test set | preference diversity | Helped |
| Increase the Dirichlet concentration parameter α (ablation) | PersonalLLM simulated users · PersonalLLM test set | preference diversity | Hurt |
| Set α too low (α=0.01) (ablation) | PersonalLLM simulated users · PersonalLLM test set | preference diversity | Hurt |
| Use only winning examples rather than zero-shot prompting | Llama3-8B-Instruct · PersonalLLM personalization test | personal preference reward | Helped |
| Add in-context examples, including incorrectly labeled examples, instead of using zero-shot prompting | Llama3-8B-Instruct · PersonalLLM personalization test | personal preference reward | Helped |
| Label losing responses as losing rather than as winning (ablation) | Llama3-8B-Instruct · PersonalLLM personalization test | personal preference reward | Helped |
| Retrieve historical users with the strongest user embedding based on pairwise preference feedback | PersonalLLM meta-learning personalization · PersonalLLM personalization test | personal preference reward | Helped |
| Use historical-user meta-learning instead of Self-ICL | PersonalLLM personalization · PersonalLLM personalization test | personal preference reward | Helped |
| Use subjective prompt cues rather than objective question cues | PersonalLLM simulated users · Prompts with subjective cues versus who/when/where prompts | population preference entropy | Helped |
| Simulate users with PersonalLLM instead of using LLMs from the OpinionQA comparison | OpinionQA preference representativeness · 59 of 60 demographic groups | representativeness score | Helped |
Surprising findings
- Including losing responses as in-context examples did not help beyond winning-only examples, and mislabeled losing responses still beat zero-shot.
- Persona prompting produces preferences only about half as diverse as reward-model-ensemble users across metrics.
Limitations
- Simulated personas are weighted reward models and may not realistically reflect human preferences; not validated against real individual users beyond OpinionQA representativeness.
- Prompts and responses were not manually inspected for quality or safety.
- Meta-learning gains over self-ICL are small with the naive embedding method.
Experimental setup
Datasets: Anthropic HH-RLHF helpful-base Anthropic HH-RLHF helpful-online HelpSteer RewardBench PersonaHub OpinionQA MTEB
Benchmarks: PersonalLLM test set OpinionQA
Metrics: reward score under simulated user preference model win rate vs GPT-4o preference diversity (majority vote share) representativeness score (1 - Wasserstein distance)
Baselines: persona prompting zero-shot self-ICL Winning only ICL Winning and Losing ICL Losing only ICL
Builds on: RewardBench persona prompting OpinionQA
Closest papers by meaning
- Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs (ICLR 2025 · similarity 0.92)
- PersonaVLM: Long-Term Personalized Multimodal LLMs (CVPR 2026 · similarity 0.91)
- Aligning LLMs by Predicting Preferences from User Writing Samples (ICML 2025 · similarity 0.90)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.89)
- HYDRA: Model Factorization Framework for Black-Box LLM Personalization (NeurIPS 2024 · similarity 0.89)
- Eliciting Human Preferences with Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.