Research map / Large language models / Alignment and preferences
Eliciting Human Preferences with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces generative active task elicitation (GATE), where LMs (GPT-4) interactively ask questions or generate edge cases to elicit user preferences, and tests it in preregistered human studies across content recommendation, moral reasoning, and email validation.
Problem
Specifying nuanced, user-dependent task preferences via prompts or labeled examples is hard because users struggle to articulate preferences and anticipate edge cases, leading to task ambiguity.
Approach
Use an LM as the elicitation policy: it interactively generates informative edge cases, yes/no questions, or open-ended questions for the user, then conditions on the transcript to predict the user's labels on held-out test items.
Key results
- GATE methods generally outperform no elicitation across all three domains; significant at 0.05 for all but the email domain, where only generative active learning was significant.
- GATE methods improved over user-written prompts in 6/10 settings for absolute p(correct) and 7/10 for AUC; generative yes/no questions improved over prompts in every setting.
- Users rated interactive GATE methods as less mentally demanding than writing prompts, especially labeling and yes/no question methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GATE elicitation instead of no elicitation | GPT-4 personalized predictors · Email validation | area under Δp(correct)-time curve | Helped |
| Use GATE elicitation instead of no elicitation | GPT-4 personalized predictors · Content recommendation | area under Δp(correct)-time curve | Helped |
| Use GATE elicitation instead of no elicitation | GPT-4 personalized predictors · Moral reasoning | area under Δp(correct)-time curve | Helped |
| Use generative yes-or-no questions instead of user-written prompts | GPT-4 personalized predictors · Content recommendation, moral reasoning, and email validation | human-predictor alignment | Helped |
| Use GATE instead of user-written prompts | GPT-4 personalized predictors · Content recommendation, moral reasoning, and email validation | area under Δp(correct)-time curve: 6/10 | Mixed |
| Use GATE instead of user-written prompts | GPT-4 personalized predictors · Content recommendation | area under Δp(correct)-time curve | Helped |
| Use GATE instead of supervised learning | GPT-4 personalized predictors · Content recommendation | area under Δp(correct)-time curve | Helped |
| Use GATE instead of pool-based active learning | GPT-4 personalized predictors · Content recommendation | area under Δp(correct)-time curve | Helped |
| Write a user-written prompt instead of providing no elicitation | GPT-4 email-validation predictor · Email validation | human-predictor alignment | Hurt |
| Use GATE instead of baseline elicitation methods | GPT-4 personalized predictors · Content recommendation, moral reasoning, and email validation | AUROC | Helped |
| Use LM-generated elicitation instead of pool-based active learning | GPT-4 model-model experiments · Content recommendation, moral reasoning, and email validation | area under Δp(correct)-turns curve | Helped |
| Use LM-generated elicitation instead of pool-based active learning | GPT-4 model-model experiments · Content recommendation, moral reasoning, and email validation | area under ΔAUROC-turns curve | Helped |
Surprising findings
- User-written prompts slightly decreased performance relative to no elicitation in the email domain, as users' stated rules often conflicted with their own labels.
- Simulated personas with GPT-4 predicted human results in content and email domains but not in moral reasoning.
Limitations
- Relies on closed-source GPT-4 whose versions are deprecated, hindering reproducibility.
- Moral reasoning domain not reliably simulated by personas; some GATE gains not statistically significant in all domains.
- Predictions use token-probability outputs from GPT-4, a workaround since logits were unavailable.
Experimental setup
Datasets: Microsoft News Dataset (MIND) authors' manually constructed test sets for email, moral reasoning, and content recommendation
Benchmarks: content recommendation (authors' test set) moral reasoning (authors' test set) email validation (authors' test set)
Metrics: area under p(correct)-time curve change in p(correct) per minute AUROC mental demand Likert rating (NASA TLX-based)
Baselines: supervised learning pool-based active learning user-written prompts no elicitation
Builds on: active learning in-context learning persona-based simulation
Closest papers by meaning
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.91)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.89)
- Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.88)
- Aligning LLMs by Predicting Preferences from User Writing Samples (ICML 2025 · similarity 0.88)
- PersonalLLM: Tailoring LLMs to Individual Preferences (ICLR 2025 · similarity 0.88)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.