Research map / Large language models / Alignment and preferences
Diverging Preferences: When do Annotators Disagree and do Models Know?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes diverging annotator preferences in MultiPref and HelpSteer2, builds a taxonomy of disagreement causes, and proposes distributional (Mean-Var, classification) reward models plus divisive-prompt detection for LLM-as-Judge benchmarks.
Problem
Standard reward models and LLM-as-Judge evaluations treat annotator disagreement as noise and make decisive preference predictions, which undermines pluralistic alignment when user preferences genuinely diverge.
Approach
Release per-annotator labels for two preference datasets, build a taxonomy of disagreement causes, and model rewards as distributions (mean and variance, trained with KL loss over all annotations) so that divisive responses can be identified and removed from benchmarks.
Key results
- Diverging preferences occur in about 39% of MultiPref and 24% of HelpSteer2 pairs; over 75% of disagreements are attributed to factors like complexity, verbosity, or underspecification.
- Single-value reward models score Diverging ID AUROC near or below chance (0.44-0.49), while Mean-Var (KL) reaches 0.615 on MultiPref and Classification (KL) reaches 0.648 on HelpSteer2.
- LLM-as-Judge picks a winner on diverging-preference pairs at rates close to high-agreement pairs (e.g., 73.8% vs 73.1% on MultiPref), and 100% of comply-vs-refuse disagreements in top divisive WildBench prompts favor the complying response.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model reward distributions and train Mean-Var with KL loss instead of independent Mean-Var with NLL | Reward models on MultiPref · MultiPref | Diverging ID AUROC: 0.549 -> 0.615 | Helped |
| Model reward distributions and train Mean-Var with KL loss instead of independent Mean-Var with NLL | Reward models on HelpSteer2 · HelpSteer2 | Diverging ID AUROC: 0.573 -> 0.582 | Helped |
| Use classification-based KL reward modeling rather than Mean-Var KL reward modeling | Reward models on HelpSteer2 · HelpSteer2 | Diverging ID AUROC: 0.582 -> 0.648 | Helped |
| Use a single-value reward model rather than a distributional reward model | Reward models on MultiPref and HelpSteer2 · MultiPref and HelpSteer2 | Diverging ID AUROC: Bradley-Terry: 0.438, 0.458, 0.482, 0.489; random: 0.5 | Hurt |
| Train Bradley-Terry on all annotator labels instead of aggregated labels | Bradley-Terry reward models · MultiPref and HelpSteer2 | Diverging ID AUROC: MultiPref: 0.458 -> 0.438; HelpSteer2: 0.482 -> 0.489 | No clear effect |
| Use a logistic CDF approximation rather than the base normal CDF approximation (ablation) | Mean-Var reward model training · Reward model training | Training stability | Helped |
| Use an overton response rather than asking for clarification | GPT-4o LLM-as-Judge single-response scoring on CocoNot underspecified prompts · CocoNot underspecified prompts | Average score: 6.94 -> 8.48 out of 10 | Helped |
| Comply rather than refuse | LLM-as-Judge on MultiPref Comply vs. Refuse disagreements · MultiPref Anthropic Harmless prompts | Win rate: 0.68 Win; 0.18 Loss; 0.13 Tie | Helped |
| Use refusals that encourage professional help rather than refusals that state inability to comply | LLM-as-Judge on MultiPref Refuse vs. Refuse disagreements · MultiPref Anthropic Harmless prompts | Win rate: 0.77 Win; 0.12 Loss; 0.12 Tie | Helped |
| Use refusals that prescribe a solution rather than refusals that state inability to comply | LLM-as-Judge on MultiPref Refuse vs. Refuse disagreements · MultiPref Anthropic Harmless prompts | Win rate: 0.74 Win; 0.22 Loss; 0.04 Tie | Helped |
| Ask for clarification rather than comply with an underspecified WildBench prompt | WildBench LLM-as-Judge · WildBench top 5% most divisive examples | Loss rate: 83% (5 total) | Hurt |
| Use distributional rewards to identify and filter divisive examples | WildBench evaluation benchmark · WildBench top 5% most divisive examples | Comply vs. Refuse disagreement share: 42% (21 total) | Helped |
Surprising findings
- Single-value reward models trained on all annotator labels still predict decisive reward gaps for diverging preferences, performing slightly worse than random at identifying them.
- LLM-as-Judge evaluations prefer overton responses over clarifying questions for underspecified prompts (8.48 vs 6.94), indicating bias toward one response strategy.
Limitations
- Taxonomy derived from manual analysis of 100 working examples per dataset with moderate inter-annotator agreement (kappa 0.58-0.59).
- Reward model experiments use only 500 held-out test examples and a single base model (Llama-3-8B-Instruct).
- Divisive-prompt analysis on WildBench covers only the top 5% (50 examples) and relies on GPT-4o-based labeling.
Experimental setup
Datasets: MultiPref HelpSteer2 Anthropic Harmless CocoNot WildBench
Benchmarks: MultiPref HelpSteer2 WildBench CocoNot
Metrics: preference accuracy Diverging ID AUROC P(Chosen > Rejected) win/loss/tie rate
Baselines: Bradley-Terry (Agg) Bradley-Terry (All) MSE Regression Skywork-Reward-Gemma-2-27B Llama-3.1-Nemotron-70B-Reward Mean-Var (NLL, Indep.)
Builds on: Bradley-Terry RLHF Arena-Hard RewardBench WildBench
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- LEGEND: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets (AAAI 2025 · similarity 0.89)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.88)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.88)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.88)
- Pairwise Calibrated Rewards for Pluralistic Alignment (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.