Research map / Large language models / Alignment and preferences

Preference Learning Algorithms Do Not Learn Preference Rankings

NeurIPS 2024 · Large language models › Alignment and preferences · approach: DPO design choices · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that preference-tuned LLMs (RLHF/DPO) achieve low likelihood-based ranking accuracy (<60%) on common preference datasets, derives an idealized ranking accuracy showing an alignment gap, and attributes it to the reference-model log-ratio term in DPO.

Problem

Preference learning (RLHF, DPO) is assumed to make models assign higher likelihood to preferred than dispreferred outputs, but it is unclear whether this ranking accuracy is actually achieved, even on training data.

Approach

Derives a closed-form idealized ranking accuracy for perfectly optimized DPO/RLHF objectives under Bradley-Terry preferences, measures the gap between it and observed ranking accuracy, and shows via a theorem that the reference model log-ratio makes mild reference ranking errors nearly impossible for DPO to flip; also relates ranking accuracy to win rate via a gamma-scaled DPO variant.

Key results

• Most preference-tuned models (e.g., Zephyr-7B-DPO 54%, Tulu-2-DPO-7B 53%, Gemma-7B-IT 54%, Llama-2-7B-Chat 53%) achieve length-normalized ranking accuracy below 60% on AlpacaFarm validation.
• Idealized ranking accuracy is often far higher, yielding alignment gaps of 19 to 59 percentage points relative to the median idealized accuracy.
• For Pythia-2.8B on HH-RLHF, fewer than 10% of initially incorrectly ranked training pairs are flipped to correct at the lowest validation loss checkpoint.

Measured effects reported in the paper

Surprising findings

• DPO rarely flips incorrectly ranked pairs before overfitting, and mostly increases reward margin of already correctly ranked pairs.
• Ranking accuracy and win rate are correlated early in training but become anti-correlated as the model moves far from the reference model.

Limitations

• Experiments on preference-tuned models are limited to a single dataset due to compute constraints (three model types, three seeds).
• Analysis is optimization-agnostic and only describes model behavior on the training preference dataset, not generalization.
• Idealized ranking accuracy requires alpha values and beta, which are unavailable for many datasets; the alignment gap was measured on AlpacaFarm validation only.

Experimental setup

Datasets: Anthropic HH-RLHF Synthetic Instruct GPT-J Pairwise StackExchange Preferences UltraFeedback Stanford Human Preferences AlpacaFarm validation

Benchmarks: AlpacaFarm validation Anthropic HH-RLHF Synthetic Instruct GPT-J Pairwise StackExchange Preferences UltraFeedback Stanford Human Preferences

Metrics: ranking accuracy length-normalized ranking accuracy idealized ranking accuracy alignment gap win rate

Baselines: Zephyr-7B-DPO Tulu-2-DPO-7B Gemma-7B-IT LLaMA-2-7B-Chat

Builds on: DPO RLHF IPO Bradley-Terry model

Closest papers by meaning

• Preference learning made easy: Everything should be understood through win rate (ICML 2025 · similarity 0.90)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (ICML 2024 · similarity 0.89)
• ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.89)
• Diverse Preference Learning for Capabilities and Alignment (ICLR 2025 · similarity 0.88)
• Group Robust Preference Optimization in Reward-free RLHF (NeurIPS 2024 · similarity 0.88)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.87)