Research map / Large language models / Alignment and preferences
Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that ratings and rankings feedback disagree on about 60% of comparisons for humans and GPT-3.5-Turbo, and that the feedback protocol used for evaluation changes which aligned model (Best-of-64 with ratings vs rankings reward models) appears better.
Problem
Sparse feedback for LLM alignment can be collected as absolute ratings or pairwise rankings, but the effect of this structural choice on feedback consistency, alignment, and evaluation is under-explored.
Approach
Converts ratings into ranking form and measures agreement with direct pairwise rankings from both humans and AI, then trains ratings-based and rankings-based reward models for Best-of-64 and shows that the evaluation protocol (ratings vs rankings) can reverse which aligned policy is preferred.
Key results
- Ratings and rankings disagree on about 58% (human) and 59.4% (GPT-3.5-Turbo) of comparisons, i.e., consistency of roughly 40-42%.
- Human-judged win-rate gap over SFT is 11.2% for Best-of-64 (rankings) vs 5.3% for Best-of-64 (ratings) under rankings evaluation, but ratings evaluation shows 5% vs 4.3%, reversing the preference.
- Best-of-64 with rankings reward reaches 45.1% win-rate vs DaVinci-003 under human pairwise judgments, versus 33.9% for SFT and 39.2% for ratings-based reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the Best-of-64 reward model with rankings feedback instead of ratings feedback | Alpaca-7B Best-of-64 · 553 unseen instructions, human rankings vs DaVinci-003 | win-rate: 39.2% -> 45.1% | Helped |
| Use Best-of-64 with a ratings-trained reward model instead of randomly sampling an SFT response | Alpaca-7B · 553 unseen instructions, human rankings vs DaVinci-003 | win-rate: 33.9% -> 39.2% | Helped |
| Use Best-of-64 with a rankings-trained reward model instead of randomly sampling an SFT response | Alpaca-7B · 553 unseen instructions, human rankings vs DaVinci-003 | win-rate: 33.9% -> 45.1% | Helped |
| Use rankings-trained rather than ratings-trained alignment | Alpaca-7B Best-of-64 · Human rankings evaluation | win-rate gap over SFT: 5.3% -> 11.2% | Helped |
| Use ratings-trained rather than rankings-trained alignment | Alpaca-7B Best-of-64 · Human ratings evaluation | win-rate gap over SFT: 4.3% -> 5% | Helped |
| Compare rankings feedback with ratings converted into rankings | GPT-3.5-Turbo feedback · 42K comparisons | feedback consistency: 42% consistent; 60% disagreement | Hurt |
| Compare rankings feedback with ratings converted into rankings | Human feedback · 500 comparisons | feedback consistency: 40.6% consistent | Hurt |
| Collect pairwise ratings rather than ratings for responses individually | Human feedback · 500 comparisons | feedback inconsistency: 58% | Hurt |
| Use a different batch of human annotators for the consistency analysis (ablation) | Human feedback consistency analysis · Human annotator splits | standard deviation of inconsistency: 1.6% | No clear effect |
| Vary GPT-3.5-Turbo temperature for feedback generation (ablation) | AI feedback consistency analysis · 1000 comparisons | standard deviation in disagreements: 2.7% | No clear effect |
| Use GPT-3.5-Turbo-0613 or GPT-4 instead of default GPT-3.5-Turbo (ablation) | AI feedback consistency analysis · Feedback protocol consistency | inconsistency: 58% -> 54%; 58% -> 50% | No clear effect |
| Convert individual ratings into rankings and compare against direct rankings | Feedback consistency analysis · Consistent vs inconsistent examples | average ratings difference: 1.58 vs 1.28 (Human); 1.22 vs 1.12 (GPT-3.5) | Mixed |
Surprising findings
- Evaluation inconsistency: the evaluation protocol (ratings vs rankings) favors the policy trained with the same protocol, reversing which aligned model appears better.
- GPT-4 shows lower inconsistency (50%) than GPT-3.5-Turbo (58%) in the appendix analysis, while all tested AI annotators still show inconsistency.
Limitations
- Evaluation focuses on win-rate against DaVinci-003 and does not measure helpfulness, harmlessness, or hallucination.
- Human annotators are Amazon Mechanical Turk workers and may not be representative; human data is small (500 rankings, 1000 ratings instances).
- Only ratings and rankings protocols are studied; richer feedback forms are not explored.
Experimental setup
Datasets: Dolly Self-Instruct (user oriented) Super-NI OpenAssistant (OASST) evaluation Anthropic helpful evaluation Vicuna evaluation Koala evaluation AlpacaEval
Benchmarks: OASST Anthropic helpful Vicuna Koala AlpacaEval
Metrics: win-rate feedback consistency (agreement rate)
Baselines: SFT (Alpaca-7B) DaVinci-003 (reference)
Builds on: RLHF RLAIF LoRA AlpacaFarm
Closest papers by meaning
- Human Feedback is not Gold Standard (ICLR 2024 · similarity 0.93)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.92)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.92)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.92)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.91)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.