Research map / Large language models / Alignment and preferences
Capturing Individual Human Preferences with Reward Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalises user-aware reward model learning with a PAC bound over raters and examples, and proposes the reward-feature model (RFM), a shared learned feature map with per-user linear weights adapted by logistic regression, evaluated on Gemma 1.1 2B with UltraFeedback and synthetic raters.
Problem
Standard RLHF reward models aggregate preferences into one user-agnostic reward, which fails under heterogeneous preferences and does not generalise to users outside the rater pool.
Approach
Model individual preferences as a linear combination of shared learned reward features; train the features on rater-labelled pairs and adapt to a new user by fitting only their linear coefficients (a convex logistic regression) on a few ranked examples.
Key results
- RFM correctly predicts held-out users' preferences around 70% of the time with 10 adaptation examples, while the non-adaptive baseline and linear/non-linear adaptive baselines perform worse and in-context LLMs fall to chance.
- RFM(32) intra-user generalisation of 61.61% on UltraFeedback is on par with VPL's reported 61.49%.
- RFM performance is much more robust to the homogeneity parameter p, with almost no degradation for d >= 32 learned features; it adapts with as few as 30 examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an adaptive reward-feature model instead of a non-adaptive reward model | Gemma 1.1 2B reward model · Synthetic raters, held-out users | accuracy | Helped |
| Increase the number of training raters | RFM · Synthetic raters, held-out users | accuracy | Helped |
| Increase preference homogeneity | RFM and non-adaptive baseline · Synthetic raters, held-out users | accuracy | Helped |
| Use RFM with at least 32 learned features as preference homogeneity decreases | RFM · Synthetic raters, held-out users | accuracy | Helped |
| Increase adaptation examples | RFM with 32 or 128 features · Synthetic raters, held-out users | accuracy | Helped |
| Use adapted RFM(32) rather than the baseline to select best-of-n responses | Gemma 2 9B and Gemma 2 27B response candidates · UltraFeedback test set, best-of-n | preference win fraction | Helped |
| Increase the number of candidate responses n | RFM(32) best-of-n selection · UltraFeedback test set | preference win fraction | Helped |
| Adapt the baseline's final layer using user-specific examples | Linear baseline · Synthetic raters, held-out users | accuracy | No clear effect |
| Use a non-linear adaptive MLP with 3 or 5 hidden layers | Gemma 1.1 2B reward model · Synthetic raters, held-out users | accuracy | Hurt |
| Use in-context personalization with 10 examples | Gemini 1.5 Pro and GPT-4o · Synthetic raters, held-out users | accuracy | Hurt |
| Use RFM(32) rather than VPL | Reward model · UltraFeedback, intra-user generalisation | accuracy: 61.49% vs 61.61% | No clear effect |
| Use RFM rather than the non-adaptive or linear baseline | Reward model · PersonalLLM and UltraFeedback, held-out reward model | accuracy | Helped |
Surprising findings
- In-context adaptation with Gemini 1.5 Pro and GPT-4o performs at chance level for predicting held-out users' preferences.
- A linear adaptation of the baseline's final layer performs on par with the non-adaptive baseline.
Limitations
- Main experiments use synthetic raters with 13 handcrafted features, which may not reflect real human disagreement.
- Real-user experiments use reward models as proxies for raters, which smooth out heterogeneity.
- Training is expensive and centralised while adaptation is cheap; the trade-off may not suit other settings.
Experimental setup
Datasets: UltraFeedback PersonalLLM
Benchmarks: UltraFeedback PersonalLLM
Metrics: inter-user accuracy on held-out users intra-user accuracy best-of-n win rate accuracy vs number of raters m and homogeneity p
Baselines: non-adaptive reward model linear adaptation of final layer MLP adapter (Park et al. 2024) Gemini 1.5 Pro in-context GPT-4o in-context VPL
Builds on: Bradley-Terry model empirical risk minimisation adapters variational preference learning
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.93)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.92)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.91)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.91)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.