Research map / Large language models / Alignment and preferences
Human Feedback is not Gold Standard
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyses whether human preference scores capture key error criteria (notably factuality), shows annotator judgments are confounded by output assertiveness and complexity, and offers preliminary evidence that RLHF-style preference training raises assertiveness.
Problem
Human preference scores are the de facto standard for evaluating and training LLMs, but it is unclear which output properties they capture and whether annotators are biased by confounders such as assertiveness and complexity.
Approach
Defines ten task-agnostic error types, compares error coverage of overall preference scores with error annotations, and uses hidden preambles to generate outputs varying in assertiveness and complexity to test annotator biases; also compares an RLHF-trained model (Llama 2 13B Chat) with a preference-tuned Command model on assertiveness vs quality.
Key results
- Overall preference scores are most strongly driven by refusal, while factuality and inconsistency contribute much less, indicating preference scores under-represent these errors.
- The assertive-- preamble raises refusal errors from 3.5% to 24%; crowdworkers underestimate factuality and inconsistency errors more for assertive outputs.
- Assertiveness correlates with overall quality at Pearson 0.68 (complexity 0.53), and Llama 2 13B Chat shows higher assertiveness for equivalent quality than Command 52B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a high-assertiveness preamble (ablation) | LLM outputs and crowdsourced error annotation · Confounder experiment | factuality error detection relative to expert annotation: -16.1 -> -22.3 | Hurt |
| Use a high-assertiveness preamble (ablation) | LLM outputs and crowdsourced error annotation · Confounder experiment | inconsistency error detection relative to expert annotation: -10.3 -> -16.7 | Hurt |
| Use a low-assertiveness preamble (ablation) | LLM outputs · Confounder experiment | refusal error rate: 3.5% -> 24% | Hurt |
| Use more complex outputs (ablation) | LLM outputs and crowdsourced error annotation · Confounder experiment | perceived repetition errors | Helped |
| Use more complex outputs (ablation) | LLM outputs and crowdsourced error annotation · Confounder experiment | crowdworker-estimated factuality errors | No clear effect |
| Use more complex outputs (ablation) | LLM outputs and expert annotation · Confounder experiment | factuality errors | Helped |
| Use preference-trained RLHF rather than preference fine-tuning | Llama 2 13B compared with Command models · Outputs under assertiveness preambles | assertiveness at equivalent quality | Hurt |
| Rate errors before assigning overall quality | Human annotators · Collected model outputs | correlation between error criteria and overall quality scores | Mixed |
| Use overall preference scores as the sole quality measure | Human evaluation · Collected model outputs | coverage of factuality and inconsistency errors | Hurt |
| Use high assertiveness (ablation) | LLM outputs and crowdsourced error annotation · Confounder experiment | crowdsourced factuality error rate: 4.3 -> 2.2 | Mixed |
Surprising findings
- Annotators underestimated factuality errors and were more trusting of assertive outputs, with the gap widening for assertive responses.
- Reference outputs received the lowest overall quality scores among the five systems evaluated.
Limitations
- Expert annotations come from the authors on only 300 examples per error type and are not strictly unbiased.
- The RLHF vs off-policy comparison is confounded because the models' training details are not directly comparable; evidence is preliminary.
- Analysis is limited to the annotation process and does not cover reward models or learning algorithms.
Experimental setup
Datasets: Curation Corpus Amazon Product Descriptions WikiHow
Benchmarks: Curation Corpus Amazon Product Descriptions WikiHow
Metrics: Pearson correlation Lasso regression weights Gwet's AC1 annotated error rate overall quality score
Baselines: Command 52B Command 6B Falcon 40B Instruct MPT 30B Instruct Llama 2 13B Chat reference outputs
Builds on: RankME Grice's Maxims RLHF
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.93)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.91)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.90)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.89)
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.88)
- The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.