Research map / Large language models / Alignment and preferences

Maximizing Signal in Human-Model Preference Alignment

AAAI 2025 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position-plus-case-study paper arguing that human-preference data should separate annotator noise from meaningful disagreement (signal) for training and evaluating LLM guardrail classifiers, illustrated with a two-classifier human alignment study.

Problem

Automatic metrics and majority-vote labeling fail to capture user preferences on subjective tasks; annotator disagreement is often discarded as noise even when it reflects legitimate value differences that models should align with.

Approach

Classify labeling tasks on a subjectivity scale (manifest, latent pattern, latent projective) to decide whether disagreement is noise to be minimized or signal to be preserved, and recommend best practices (agreement statistics, adequate sample size, representative sampling, soft metrics) for each case.

Key results

• The two guardrail classifiers showed very low agreement (Cohen's kappa = 0.0937) on 7795 PRISM questions, with a significant McNemar difference (chi2 = 919.18, p < .001).
• In a crowd-sourced survey (80 participants, 50 items), human raters showed slight-to-fair agreement (Krippendorff's alpha = 0.21).
• Model 2 was chosen as the more conservative classifier, flagging 988 questions that Model 1 did not, aligning with participant judgments that most items required subjective or value-based responses.

Measured effects reported in the paper

Surprising findings

• Participant ratings skewed toward 'Yes' (subjective/value-based), and participants were judged conservative, which motivated the choice of the more conservative classifier.

Limitations

• Methodology requires researcher knowledge of end-user population properties.
• Crowd-sourcing platforms (Prolific) are not representative of country-wide populations.
• Case study uses a single company feature with a small, filtered 50-item survey sample.

Experimental setup

Datasets: PRISM

Benchmarks: PRISM controversial question set (7795 labeled items)

Metrics: Cohen's kappa Krippendorff's alpha McNemar's chi-squared test

Baselines: BLEU ROUGE METEOR BERTScore

Builds on: RLHF Constitutional AI

Closest papers by meaning

• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.91)
• Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.91)
• Human Feedback is not Gold Standard (ICLR 2024 · similarity 0.89)
• Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.89)
• Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.89)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)