Research map / Reinforcement learning / RL from feedback
PILAF: Optimal Human Preference Sampling for Reward Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PILAF, a preference-data sampling scheme that interpolates/extrapolates the policy and reference logits to generate response pairs, aligning the DPO/MLE gradient with oracle-value optimization, and validates it in iterative and online DPO on HH-RLHF.
Problem
RLHF reward modeling and DPO use preference data sampled uniformly from the current policy, which misaligns the MLE/DPO gradient with maximizing the oracle objective; the question is how to sample response pairs for human labeling to fix this.
Approach
Generate response pairs from policies tilted by the implicit reward (pi_theta * exp(+r) and pi_theta * exp(-r)), approximated token-wise as logit interpolation/extrapolation between the policy and reference (PILAF), so the MLE gradient matches the oracle-objective gradient to first order and the covariance aligns with the Hessian of J.
Key results
- Iterative DPO: PILAF reward -9.80, KL 25.01, J -12.30, best J among methods (Vanilla J -13.68, Hybrid J -12.80), reaching baseline end-point reward about halfway through training (~40% less training time).
- Online DPO: PILAF reward -4.88, KL 15.42, J -6.42, versus Vanilla reward -4.96 with KL 21.50 and Best-of-N J -6.77.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PILAF sampling instead of Best-of-N sampling | Iterative DPO · HH-RLHF test set | reward: -10.13 -> -9.80 | Helped |
| Use PILAF sampling instead of Hybrid sampling | Iterative DPO · HH-RLHF test set | KL divergence: 22.86 -> 25.01 | Hurt |
| Use PILAF sampling instead of Hybrid sampling | Iterative DPO · HH-RLHF test set | J objective: -12.80 -> -12.30 | Helped |
| Use PILAF sampling instead of Vanilla sampling | Online DPO · HH-RLHF test set | reward: -4.96 -> -4.88 | Helped |
| Use PILAF sampling instead of Best-of-N sampling | Online DPO · HH-RLHF test set | KL divergence: 12.35 -> 15.42 | Hurt |
| Use PILAF sampling instead of Best-of-N sampling | Online DPO · HH-RLHF test set | J objective: -6.77 -> -6.42 | Helped |
Surprising findings
- Vanilla online DPO reaches roughly the same reward as PILAF but at substantially higher KL divergence.
- Best-of-N with DPO reward has the lowest KL in online DPO (12.35) yet lower reward than PILAF.
Limitations
- Experiments limited to 8B models and a reward model simulating human feedback rather than real human labels.
- Approximations: partition functions replaced by 1 and token-wise factorization, justified only for small beta.
- Theory relies on second-order approximations and asymptotic/regularity assumptions.
Experimental setup
Datasets: HH-RLHF
Benchmarks: HH-RLHF test set
Metrics: average reward (oracle proxy) KL divergence from reference objective J
Baselines: Vanilla sampling Best-of-N sampling with DPO reward Hybrid sampling (policy vs reference)
Builds on: DPO PPO iterative DPO online DPO Bradley-Terry model
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.92)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.92)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.92)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.92)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.