Research map / Large language models / Alignment and preferences
Preference Ranking Optimization for Human Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Preference Ranking Optimization (PRO), an SFT-style algorithm that extends pairwise Bradley-Terry contrasts to iterative one-to-N contrasts over human preference rankings of any length, optionally with reward-model-based dynamic temperatures and self-bootstrapped candidates.
Problem
RLHF for LLM human alignment is complex, unstable, and hyperparameter-sensitive, and pairwise contrasts discard most of the information in multi-response preference rankings.
Approach
Recursively contrast the top remaining response against all lower-ranked ones (one-to-N, InfoNCE-style, Plackett-Luce-like) so the LLM's length-normalized log-likelihoods align with a full human preference ranking, trained purely with SFT-style objectives plus an NLL term; a reward model can rank expanded candidate sets and set dynamic temperatures.
Key results
- On HH-RLHF raw (ranking length 2), PRO reaches a total reward of 55.35 versus 48.83 for SFT and 52.75 for DPO, with BLEU 21.54.
- With ChatGPT-expanded rankings of length 3, PRO reaches a total reward of 67.97 and is competitive with ChatGPT zero-shot (68.48) using a 7B model.
- In GPT-4 evaluation, PRO beats the golden human responses with an average win rate of 55.00% vs 40.63% lose; human annotators gave 22.50% win, 56.25% tie, 21.25% lose.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PRO instead of DPO | LLaMA-7B on HH-RLHF raw · HH-RLHF raw (Total) | Reward: 52.75 -> 55.35 | Helped |
| Use PRO instead of DPO | LLaMA-7B on HH-RLHF Alpaca, 3 candidates · HH-RLHF Alpaca, 3 candidates (Total) | Reward: 59.27 -> 58.72 | Hurt |
| Use PRO instead of DPO | LLaMA-7B on HH-RLHF ChatGPT, 3 candidates · HH-RLHF ChatGPT, 3 candidates (Total) | Reward: 64.10 -> 67.97 | Helped |
| Add the SFT loss to PRO (ablation) | LLaMA-7B on HH-RLHF raw · HH-RLHF raw (Total) | BLEU: 9.85 -> 21.54 | Helped |
| Add the SFT loss to PRO (ablation) | LLaMA-7B on HH-RLHF Alpaca, 3 candidates · HH-RLHF Alpaca, 3 candidates (Total) | BLEU: 18.29 -> 22.11 | Helped |
| Add the recursive ranking losses for positions k>1 (ablation) | LLaMA-7B on HH-RLHF Alpaca, 3 candidates · HH-RLHF Alpaca, 3 candidates (Total) | Reward: 58.11 -> 58.72 | Helped |
| Add the recursive ranking losses for positions k>1 (ablation) | LLaMA-7B on HH-RLHF ChatGPT, 3 candidates · HH-RLHF ChatGPT, 3 candidates (Total) | Reward: 67.75 -> 67.97 | Helped |
| Keep dynamic temperature (ablation) | LLaMA-7B on HH-RLHF raw · HH-RLHF raw (Total) | Reward: 55.04 -> 55.35 | Helped |
| Keep dynamic temperature (ablation) | LLaMA-7B on HH-RLHF Alpaca, 3 candidates · HH-RLHF Alpaca, 3 candidates (Total) | Reward: 58.40 -> 58.72 | Helped |
| Keep dynamic temperature (ablation) | LLaMA-7B on HH-RLHF ChatGPT, 3 candidates · HH-RLHF ChatGPT, 3 candidates (Total) | Reward: 68.40 -> 67.97 | Hurt |
| Keep both SFT loss and dynamic temperature (ablation) | LLaMA-7B on HH-RLHF raw · HH-RLHF raw (Total) | Reward: 46.17 -> 55.35 | Helped |
| Add a self-generated response to the ranked candidates (ablation) | PRO on HH-RLHF raw · HH-RLHF raw (Total) | BLEU: 21.54 -> 23.77 | Helped |
Surprising findings
- Naive self-bootstrapping gave conflicting results: it helped only on the ChatGPT-expanded set and hurt on the Alpaca-expanded and raw sets, which the authors attribute to the weak underlying model and 1.4B reward model.
- Adding lower-quality responses (Curie and Alpaca) to a ranking sometimes outperformed using only higher-quality Alpaca responses.
Limitations
- Evaluation relies on reward models and GPT-4 judgments; human evaluation covers a small sample (20 items per subset).
- Self-bootstrapping results are inconsistent and may depend on a small 1.4B reward model and a 7B policy.
- Experiments use only LLaMA-7B and a single dataset (HH-RLHF).
Experimental setup
Datasets: HH-RLHF
Benchmarks: HH-RLHF
Metrics: BLEU reward score win/tie/lose rate
Baselines: SFT BoN RLHF CoH DPO RRHF ChatGPT ChatGLM
Builds on: RLHF Bradley-Terry model InfoNCE Plackett-Luce model DPO
Closest papers by meaning
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.93)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.93)
- HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.93)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.92)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.