Research map / Large language models / Alignment and preferences
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows best-of-n is essentially win-rate-vs-KL optimal among reward-tilted policies, and proposes BoNBoN alignment, a loss combining SFT and IPO on best/worst-of-n pairs to distill best-of-n into a single model without inference-time sampling.
Problem
Best-of-n sampling is effective for LLM alignment but costs n samples per inference, and it is unclear how it compares to RLHF/contrastive alignment methods in win-rate versus KL trade-off and off-target drift.
Approach
Embeds best-of-n and RLHF/DPO policies in a common class of reward-tilted distributions, showing best-of-n nearly attains the optimal win-rate-vs-KL frontier; then trains a single LLM to mimic the best-of-n distribution via a combined SFT loss on best-of-n samples and an IPO-style loss on best-vs-worst pairs with an analytically derived beta, with no KL regularization toward the reference.
Key results
- Best-of-n achieves win rate n/(n+1) against the reference with KL log(n)-(n-1)/n, and the max win-rate gap to the optimal KL-tilted policy is under 1 percentage point at n=2.
- BoNBoN reaches around 85% win rate with much smaller response-length change than DPO/IPO baselines, which nearly double average length at comparable win rates.
- BoNBoN outperforms SFT-BoN and IPO-BoN when mimicking best-of-n across beta values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use best-of-n sampling instead of an arbitrary reward-tilted alignment policy | Alignment policy · Win rate versus KL frontier | win rate: less than 1 percentage point | Helped |
| Combine SFT-BoN and IPO-BoN objectives | BoNBoN · Anthropic HH and OpenAI TL;DR | win rate | Helped |
| Train on best-of-n samples only, without the IPO-BoN term (ablation) | BoNBoN · Anthropic HH and OpenAI TL;DR | win rate | Hurt |
| Use IPO-BoN alone, without the SFT-BoN term (ablation) | BoNBoN · Anthropic HH and OpenAI TL;DR | response length | Hurt |
| Train on best-of-n data rather than original preference completions | Alignment methods · Anthropic HH and OpenAI TL;DR | win rate | Helped |
| Add an SFT term to the contrastive objective | Contrastive alignment methods · Anthropic HH and OpenAI TL;DR | response length | Helped |
| Use the analytically derived IPO-BoN β instead of alternative β values (ablation) | IPO-BoN · Anthropic HH and OpenAI TL;DR | win rate versus response length | Mixed |
| Vary the BoNBoN α from the default 0.005 (ablation) | BoNBoN · Anthropic HH and OpenAI TL;DR | win rate versus response length | Mixed |
| Vary β from the analytically derived β* (ablation) | BoNBoN · Anthropic HH and OpenAI TL;DR | win rate versus response length | Mixed |
| Train DPO or IPO on best-and-worst-of-8 samples rather than original preference data | DPO and IPO · Anthropic HH and OpenAI TL;DR | win rate versus off-target deviation | Helped |
Surprising findings
- Contrastive methods (DPO/IPO) trained on best-vs-worst data tend to increase response length by pushing down likelihood of both winning and losing responses.
- Estimated KL divergence correlates poorly with salient off-target changes such as response length.
Limitations
- Theory assumes continuous rewards and a well-specified tilting; discrete-response corrections are only bounded approximately.
- Experiments use a learned reward model as ground truth and a small 2.8B model on two tasks.
- Relies on a single alpha hyperparameter to balance SFT and IPO terms.
Experimental setup
Datasets: Anthropic HH OpenAI TL;DR
Benchmarks: Anthropic HH OpenAI TL;DR
Metrics: win rate vs reference (by ground-truth reward model) estimated KL divergence average response length
Baselines: DPO IPO SFT-BoN IPO-BoN best-of-8 sampling
Builds on: RLHF DPO IPO
Closest papers by meaning
- Variational Best-of-N Alignment (ICLR 2025 · similarity 0.94)
- Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment (ICML 2025 · similarity 0.90)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.90)
- Theoretical guarantees on the best-of-n alignment policy (ICML 2025 · similarity 0.89)
- BOND: Aligning LLMs with Best-of-N Distillation (ICLR 2025 · similarity 0.89)
- Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.