Research map / Large language models / Alignment and preferences
Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that preference learning with hidden context implicitly aggregates via Borda count, and proposes distributional preference learning (DPL) with risk-averse optimization to reduce RLHF jailbreak vulnerability.
Problem
Preference learning (including RLHF reward models) is trained on data where annotator-affecting variables such as annotator identity, irrationality, or differing objectives are hidden from the learned utility model, so the learned utility implicitly aggregates over this hidden context in unclear way
Approach
Proves BTL preference learning with hidden context implicitly implements Borda count rather than expected utility, and introduces distributional preference learning that outputs a distribution of utilities per alternative, enabling detection of hidden context (via r^2) and risk-averse optimization over lower quantiles.
Key results
- Standard combined reward model has a 25.1% jailbreak rate, versus 52.4% for helpful-only and 3.7% for harmless-only models.
- Categorical DPL with risk-averse (0.01-quantile) optimization reduces jailbreak rate to 13.4% from 25.1% for the standard combined model, with helpfulness accuracy of 66.2%.
- DPL r^2 is lower on the combined dataset (0.53 mean-and-variance, 0.41 categorical) than on single-objective datasets (0.77 to 0.89), indicating detectable hidden context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the 0.01-quantile instead of the mean for Mean & var. DPL (ablation) | Mean & var. DPL · HH-RLHF jailbreak prompts | jailbreak rate: 30.5% -> 20.3% | Helped |
| Use the 0.01-quantile instead of the mean for Mean & var. DPL (ablation) | Mean & var. DPL · HH-RLHF helpfulness test set | helpfulness accuracy: 68.4% -> 66.4% | Hurt |
| Use the 0.01-quantile instead of the mean for Categorical DPL (ablation) | Categorical DPL · HH-RLHF jailbreak prompts | jailbreak rate: 32.1% -> 13.4% | Helped |
| Use the 0.01-quantile instead of the mean for Categorical DPL (ablation) | Categorical DPL · HH-RLHF helpfulness test set | helpfulness accuracy: 66.2% -> 66.2% | No clear effect |
| Train one standard reward model on the combined helpfulness and harmlessness data rather than harmlessness-only data | Standard preference learning · HH-RLHF jailbreak prompts | jailbreak rate: 3.7% -> 25.1% | Hurt |
| Train on combined objectives rather than helpfulness-only data | Standard preference learning · HH-RLHF jailbreak prompts | jailbreak rate: 52.4% -> 25.1% | Helped |
| Train DPL on combined objectives rather than Helpful-only data | Mean & var. DPL · HH-RLHF | r^2: 0.89 -> 0.53 | Mixed |
| Train DPL on combined objectives rather than Harmless-only data | Mean & var. DPL · HH-RLHF | r^2: 0.77 -> 0.53 | Mixed |
| Train DPL on combined objectives rather than Helpful-only data | Categorical DPL · HH-RLHF | r^2: 0.63 -> 0.41 | Mixed |
| Train DPL on combined objectives rather than Harmless-only data | Categorical DPL · HH-RLHF | r^2: 0.53 -> 0.41 | Mixed |
Surprising findings
- Preference learning can prefer an alternative with lower expected utility even when a majority of pairwise comparisons favor the higher-expected-utility option (Borda count disagreement with expected utility).
Limitations
- Theoretical identifiability of expected utility fails in general, so DPL does not converge to true expected utility.
- Case study limited to a single LLM preference dataset with synthetically relabeled objectives.
- Jailbreak evaluation uses a fixed set of jailbreak prompt pairs rather than adaptive attacks.
Experimental setup
Datasets: HH-RLHF jailbreak prompt pairs from prior work
Benchmarks: HH-RLHF helpfulness test set jailbreak prompt pairs
Metrics: jailbreak rate helpfulness accuracy r^2 explained variance
Baselines: Standard preference learning (helpful-only) Standard preference learning (harmless-only) Standard preference learning (combined)
Builds on: Bradley-Terry-Luce model Thurstone-Mosteller model RLHF Borda count
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
- More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.90)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.90)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.89)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.