Research map / Large language models / Alignment and preferences
BOND: Aligning LLMs with Best-of-N Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BOND, an RLHF method that distills the Best-of-N sampling distribution into a policy via online distribution matching, instantiated as J-BOND (Jeffreys divergence with EMA anchor) and applied to Gemma models.
Problem
Best-of-N sampling gives strong reward-KL trade-offs but multiplies inference cost by N; RLHF policies need a way to match BoN quality with a single sample at inference, while avoiding forgetting and reward hacking.
Approach
Derive the exact analytical Best-of-N distribution and cast alignment as distribution matching to it, minimizing a Jeffreys divergence (forward KL via SFT on best-of-N samples plus backward KL via a quantile-based policy gradient), with an iterative procedure using an EMA moving anchor and Best-of-2 targets.
Key results
- On XSum with T5, J-BOND (beta=0.5) minimizes both forward and backward KL to the Best-of-N distribution and improves reward quantiles more than the forward-KL-only variant (beta=0).
- J-BOND gives a better reward/KL Pareto front than REINFORCE across regularization strengths {0.001, 0.01, 0.1, 1} on Gemma 7B.
- Iterative BOND with n=2 or 4 keeps improving reward while non-iterative BOND with fixed N saturates early.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Jeffreys divergence (β=0.5) instead of forward KL (β=0) or backward KL (β=1) (ablation) | BOND · XSum | forward KL to Best-of-N distribution | Helped |
| Use Jeffreys divergence (β=0.5) instead of forward KL (β=0) or backward KL (β=1) (ablation) | BOND · XSum | backward KL to Best-of-N distribution | Helped |
| Use Jeffreys divergence (β=0.5) instead of forward KL (β=0) (ablation) | BOND · XSum | reward log quantiles | Helped |
| Distill Best-of-N iteratively from a moving anchor instead of using non-iterative BOND (ablation) | BOND · XSum | average reward | Helped |
| Distill Best-of-N iteratively from a moving anchor instead of using non-iterative BOND (ablation) | BOND · XSum | reward/KL trade-off | No clear effect |
| Use an EMA anchor instead of periodic anchor updates every 50 steps (ablation) | JBOND on Gemma 7B · Conversational prompts | KL from reference policy | Helped |
| Use an EMA anchor instead of periodic anchor updates every 50 steps (ablation) | JBOND on Gemma 7B · Conversational prompts | average reward | No clear effect |
| Increase the anchor mixing parameter η (ablation) | JBOND on Gemma 2B · Conversational prompts | rate of average reward increase | Helped |
| Add/increase additional KL regularization γ (ablation) | JBOND on Gemma 2B · Conversational prompts | reward/KL trade-off | Helped |
| Use a learned quantile model instead of Monte-Carlo quantile estimation (ablation) | BOND · XSum | log quantiles | No clear effect |
| Use JBOND instead of REINFORCE baselines | Gemma 7B · Conversational prompts | reward/KL trade-off | Helped |
Surprising findings
- Reported that the correction factor (B) in the BoN distribution does not play a significant role in practice, so the quantile reward is approximated by the quantile alone.
- A Monte-Carlo quantile estimate with only 2 anchor samples is too noisy, motivating a hand-designed -log(16) reward for worse-than-both samples.
Limitations
- The J-BOND reward is a crude approximation of the log quantile when only 2 anchor samples are used.
- Evaluation of the Gemma experiments relies on a large proprietary reward model and academic benchmarks not fully specified in this text.
Experimental setup
Datasets: XSum
Benchmarks: XSum
Metrics: reward KL divergence reward/KL trade-off log reward quantile
Baselines: REINFORCE Best-of-N sampling Mixtral
Builds on: REINFORCE Best-of-N sampling WARP KL-regularized RLHF
Closest papers by meaning
- BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling (NeurIPS 2024 · similarity 0.89)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.89)
- Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models (ICLR 2025 · similarity 0.89)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.89)
- Variational Best-of-N Alignment (ICLR 2025 · similarity 0.88)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.