Research map / Large language models / Alignment and preferences
HPS: Hard Preference Sampling for Human Preference Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hard Preference Sampling (HPS), a loss that prioritizes the most preferred response and rejects all dispreferred ones, weighting hard (high-reward) dispreferred responses via single-sample Monte Carlo sampling to cut PL training cost.
Problem
Plackett-Luce and Bradley-Terry preference losses handle harmful responses poorly, treat dispreferred responses indiscriminately, and PL training is computationally expensive for LLM alignment.
Approach
A softmax-style loss over the preferred response versus all dispreferred responses, reweighted by an exponential-of-reward hardness distribution (gamma-scaled) and approximated by sampling a single dispreferred response per prompt per iteration, claimed to give better sample complexity than PL and to maximize the margin to the hardest dispreferred response.
Key results
- DPO-HPS on HH-RLHF reaches RM_DPO 2.723 vs -0.795 for DPO-PL, with BLEU ~0.232 and reward 0.430, comparable to baselines.
- HPS reduces fine-tuning time from 168.4 h (PL) to 64.4 h, a 61.76% reduction.
- In the user study DPO-HPS scores 3.93 quality vs 3.82 DPO-BT, 3.69 DPO-PL, 3.63 SFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BT with HPS | DPO · HH-RLHF fine-tuning | RM_DPO: 0.349 -> 2.723 | Helped |
| Replace BT with HPS | DPO · HH-RLHF fine-tuning | BLEU: 0.230 -> 0.232 | No clear effect |
| Replace BT with HPS | DPO · PKU-SafeRLHF fine-tuning | Reward: 0.417 -> 0.407 | Hurt |
| Replace BT with HPS | DPO · PKU-SafeRLHF fine-tuning | RM_DPO: -5.441 -> -5.359 | Helped |
| Replace BT with HPS | DPO · HH-RLHF transfer | RM_DPO: -5.053 -> -4.816 | Helped |
| Replace BT with HPS | DPO · PKU-SafeRLHF transfer | RM_DPO: 1.346 -> 5.725 | Helped |
| Increase response candidates per prompt from 5 to 100 (ablation) | DPO-HPS · HH-RLHF fine-tuning | RM_DPO: 0.600 -> 2.723 | Helped |
| Increase response candidates per prompt from 5 to 100 (ablation) | DPO-HPS · HH-RLHF fine-tuning | RM_R-DPO: -0.273 -> 2.040 | Helped |
| Increase response candidates per prompt from 5 to 100 (ablation) | DPO-HPS · HH-RLHF fine-tuning | BLEU: 0.229; 0.224; 0.230; 0.232 | Mixed |
| Increase response candidates per prompt from 5 to 100 (ablation) | DPO-HPS · HH-RLHF fine-tuning | Reward: 0.431; 0.432; 0.431; 0.430 | Mixed |
Surprising findings
- BLEU and reward scores are nearly identical across PL, BT and HPS, so gains appear mainly in reward margin.
Limitations
- Win-rate evaluation relies on open-source LLM judge (Qwen2.5-72B) due to budget, not GPT-4 or Claude.
- Gains on BLEU and reward are marginal; improvements concentrated in reward-margin metrics that are defined by the same implicit reward family being optimized.
Experimental setup
Datasets: HH-RLHF PKU-SafeRLHF
Benchmarks: HH-RLHF PKU-SafeRLHF
Metrics: BLEU reward (ArmoRM-Llama3) reward margin (RM_DPO, RM_R-DPO) win rate human quality score (Likert 1-5)
Baselines: DPO-PL DPO-BT EXO IPO SPPO NCA SFT
Builds on: DPO Plackett-Luce model Bradley-Terry model IPO SPPO
Closest papers by meaning
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.93)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.91)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.91)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.