Research map / Reinforcement learning / RL from feedback
Statistical Rejection Sampling Improves Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Statistical Rejection Sampling Optimization (RSO), which samples preference pairs from an estimated optimal policy via rejection sampling guided by a pairwise reward-ranking model, and unifies DPO (logistic) and SLiC (hinge) losses for offline preference optimization.
Problem
Offline preference optimization (DPO, SLiC) fits the optimal policy on preference data not sampled from that policy, so the maximum likelihood estimate of the optimal policy is biased; the goal is to source preference pairs closer to the optimal policy.
Approach
Use statistical rejection sampling with the SFT policy as proposal and a pairwise reward-ranking model's implied reward to draw candidates from the estimated optimal policy, label sampled pairs with the reward model, then fit sigmoid or hinge losses; top-k-over-N rejection sampling is shown to be a special case.
Key results
- On Reddit TL;DR with T5-large, RSO sigmoid-norm reached 82.22% gold reward and 71.86% AutoSxS win rate versus 76.09% and 67.72% for DPO.
- On AnthropicHH with T5-large, RSO sigmoid-norm reached 40.98% AutoSxS versus 24.01% for DPO (direct).
- With T5-XXL, RSO sigmoid-norm improved AutoSxS over DPO by 1.1% on Reddit TL;DR and 33.1% on AnthropicHH.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace direct preference pairs with RSO-sample-rank pairs | DPO sigmoid-norm · Reddit TL;DR | AutoSxS (%): 67.72 -> 71.86 | Helped |
| replace SFT-sample-rank pairs with RSO-sample-rank pairs | DPO sigmoid-norm · Reddit TL;DR | AutoSxS (%): 69.02 -> 71.86 | Helped |
| replace SFT-sample-rank pairs with RSO-sample-rank pairs | DPO sigmoid-norm · AnthropicHH | AutoSxS (%): 39.56 -> 40.98 | Helped |
| replace direct preference pairs with RSO-sample-rank pairs | DPO sigmoid-norm · AnthropicHH | AutoSxS (%): 24.01 -> 40.98 | Helped |
| use tournament ranking instead of first-round ranking (ablation) | RSO with 8 sampled responses · Reddit TL;DR | Proxy Reward (%): 92.37 -> 93.35 | Helped |
| use tournament ranking instead of first-round ranking (ablation) | RSO with 8 sampled responses · Reddit TL;DR | AutoSxS (%): 71.86 -> 71.69 | Hurt |
| increase SFT-sampled candidates from 8 to 64 with first-round ranking (ablation) | SFT-sample-rank · Reddit TL;DR | AutoSxS (%): 68.51 -> 68.84 | Helped |
| use tournament ranking instead of first-round ranking (ablation) | SFT-sample-rank with 64 candidates · Reddit TL;DR | AutoSxS (%): 68.84 -> 71.08 | Helped |
| add SLiC regularization loss (ablation) | SLiC sample-rank · Reddit TL;DR | AutoSxS (%): 67.34 -> 67.64; 67.79; 67.84; 67.44 | No clear effect |
| use hinge-norm instead of hinge loss | preference optimization with RSO-sample-rank · Reddit TL;DR | AutoSxS (%) | Helped |
| vary loss temperature gamma (ablation) | RSO · Reddit TL;DR | Proxy Reward win rate | Mixed |
| vary rejection-sampling temperature beta (ablation) | RSO · Reddit TL;DR | Proxy Reward win rate | Mixed |
Surprising findings
- Hinge loss (SLiC) shows reward hacking on Reddit TL;DR, with higher proxy reward win rates but lower AutoSxS than other losses.
- Adding the SLiC regularization loss gave no significant improvement, so it was dropped.
Limitations
- Requires additional sampling (64 SFT candidates) and reward-model ranking, adding computation over DPO and SLiC-direct.
- Evaluation relies on proxy reward, a PaLM 2-based gold reward model, and PaLM 2-L AutoSxS judgments; human study is small (47 raters).
- Experiments limited to T5 models and two main tasks; reward model accuracy is only about 70-73%.
Experimental setup
Datasets: Reddit TL;DR AnthropicHH CNN/DailyMail
Benchmarks: Reddit TL;DR AnthropicHH CNN/DailyMail
Metrics: proxy reward win rate (%) gold reward win rate (%) AutoSxS win rate (%) human preference rate human quality score (1-5)
Baselines: DPO SLiC RAFT ReST
Builds on: DPO SLiC RAFT ReST
Closest papers by meaning
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.91)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.90)
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.