Research map / Large language models / Alignment and preferences
Aligning LLMs by Predicting Preferences from User Writing Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PROSE, which infers user preference descriptions from writing samples via iterative refinement and component-level consistency verification, and introduces the PLUME benchmark to replace PRELUDE.
Problem
Preference descriptions inferred by LLM writing agents from user demonstrations tend to be generic and fail to capture nuanced individual writing preferences; existing benchmark PRELUDE has weak metric correlation and overlapping preference sets.
Approach
Iteratively refine an inferred preference description by comparing the agent's generation to the user's demonstration, decompose it into preference components, and verify each component against multiple user demonstrations with an LLM judge before storing it in memory.
Key results
- PROSE_Full outperforms CIPHER by 33% on the percentile-scaled PPCM generation quality metric across the four LLMs.
- PROSE_Full+ICL gives up to 9% improvement over ICL alone, and 7.8%/8.9%/51.1% gains over PROSE, ICL and CIPHER with GPT-4o.
- In a 16-participant human A/B test, PLUME+ICL won 69.4% of comparisons against ICL and 91.8% against CIPHER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add assistant-generated comparison samples when inferring preferences (ablation) | PROSE · PLUME | PPCM: 3.8% mean improvement | Helped |
| Allow multiple preference refinement steps (ablation) | PROSE · PLUME | PPCM: 7.8% mean improvement | Helped |
| Update candidate generations after each inference step using newly inferred preferences (ablation) | PROSE · PLUME | PPCM: 3.2% improvement | Helped |
| Add iterative refinement to preference inference (ablation) | PROSE · PLUME | PPCM: 14.8% mean improvement | Helped |
| Add preference consistency verification across demonstrations (ablation) | PROSE · PLUME | Preference similarity: 1.5% | Helped |
| Add preference consistency verification across demonstrations (ablation) | PROSE · PLUME | PPCM: 1.7% | Helped |
| Combine PROSE with ICL | PROSE · PLUME with GPT-4o agent | PPCM: 7.8%, 8.9%, and 51.1% over PROSE, ICL, and CIPHER | Helped |
| Combine PROSE with ICL | PROSE · PLUME human evaluation | win rate vs ICL: 69.4% | Helped |
| Combine PROSE with ICL | PROSE · PLUME human evaluation | win rate vs CIPHER: 91.8% | Helped |
| Sort preference components by length rather than keeping the LLM's order (ablation) | PROSE · PLUME tasks | performance: 11% | Hurt |
| Use PROSE Full instead of CIPHER | Preference inference and writing generation on PLUME · PLUME | percentile performance: 33% | Helped |
Surprising findings
- ICL outperforms PROSE on average across LLMs in PPCM, with PROSE only beating ICL when using GPT-4o.
- Preference-quality metrics such as accuracy and preference similarity correlate weakly with generation quality on PRELUDE, and Levenshtein distance is poorly correlated with preference quality.
Limitations
- PROSE_Full uses roughly 6x more tokens than the single-step CIPHER-style variant.
- Consistency verification gives only modest gains, and current metrics struggle to measure the effect of pruned preferences.
- Evaluation uses GPT-4o as a synthetic human proxy; a full-scale human trial is left for future work.
Experimental setup
Datasets: PRELUDE PLUME
Benchmarks: PLUME PRELUDE
Metrics: Preference-Similarity PPCM Pref. Sim. Levenshtein distance BERTScore accuracy
Baselines: NPC Oracle ICL CIPHER-1 CIPHER-5 PROSE_CE
Builds on: CIPHER PRELUDE
Closest papers by meaning
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.90)
- PersonalLLM: Tailoring LLMs to Individual Preferences (ICLR 2025 · similarity 0.90)
- Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs (ICLR 2025 · similarity 0.89)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.89)
- Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward (NeurIPS 2025 · similarity 0.89)
- ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.