Research map / Large language models / Alignment and preferences
Variational Best-of-N Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the output distribution of Best-of-N sampling and fine-tunes a language model with PPO to minimize reverse KL to that distribution (variational BoN), recovering BoN-like quality at 1/N inference cost.
Problem
Best-of-N alignment is effective but reduces inference throughput by a factor of N; the goal is to distill its behavior into a fine-tuned model so inference costs the same as the reference model.
Approach
Derive the closed-form BoN distribution via the strict CDF of rewards under the reference model, then fine-tune the policy by minimizing reverse KL to it (variational BoN objective), using a Monte Carlo estimate of log CDF; the objective depends on rewards only through rankings, and N acts as a regularization strength analogous to 1/beta in KL-RL.
Key results
- On sentiment control, VBON appears on the Pareto front of reward vs KL more often than other fine-tuning methods (50% of Pareto points vs 33% PPO, 10% DPO, 7% BoNBoN in one reported breakdown).
- On summarization, VBON significantly outperforms PPO in win rate and reward at lower sampling temperatures (0.25 and 0.5) and is closest to BoN among fine-tuning methods.
- VBON keeps inference cost equal to the reference model while approximating BoN with N=100 in the summarization setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with the variational Best-of-N objective instead of the KL-constrained RL objective | Pythia-2.8B SFT · Reddit TL;DR, lower sampling temperatures (0.25 and 0.5) | win rate and average reward | Helped |
| Use the variational Best-of-N objective rather than KL-constrained RL | Pythia-2.8B SFT · Reddit TL;DR, higher sampling temperatures (0.75 and 1) | win rate | No clear effect |
| Use DPO rather than PPO | GPT-IMDB · Sentiment control | win rate and average reward | Mixed |
| Increase the Monte Carlo sample count M for estimating the CDF (ablation) | Variational Best-of-N · Sentiment control | reward and win rate | Helped |
| Use M = 32 Monte Carlo samples (ablation) | Variational Best-of-N · Sentiment control | reward and win rate | No clear effect |
| Replace the full variational Best-of-N objective with its L2 lower bound (ablation) | Variational Best-of-N · Sentiment control | alignment performance | No clear effect |
| Increase the sampling temperature (ablation) | Summarization alignment methods · Reddit TL;DR | average reward | Hurt |
| Increase N for the variational Best-of-N objective (ablation) | Variational Best-of-N · Sentiment control | reward–KL trade-off | Mixed |
Surprising findings
- BoN-SFT (maximum likelihood on BoN samples) fails to trade off reward and KL well and diverges too far from the reference model.
- DPO achieves higher win rates than PPO on sentiment but lower average reward.
Limitations
- Requires precomputing log CDF estimates via Monte Carlo over reference samples, with sample count M affecting performance.
- The BoN distribution derivation assumes a one-to-one reward mapping; non-injective rewards need tie-breaking.
- Lower bound L2 and approximation are only asymptotic or heuristic; experiments limited to two tasks with small models.
Experimental setup
Datasets: IMDB Reddit TL;DR
Benchmarks: IMDB sentiment control Reddit TL;DR summarization
Metrics: average reward KL divergence win rate Pareto frontier frequency
Baselines: BoN PPO DPO BoNBoN BoN-SFT
Builds on: best-of-n sampling KL-constrained RL objective PPO DPO BoNBoN
Closest papers by meaning
- BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling (NeurIPS 2024 · similarity 0.94)
- Majority of the Bests: Improving Best-of-N via Bootstrapping (NeurIPS 2025 · similarity 0.89)
- Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment (ICML 2025 · similarity 0.89)
- Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models (ICLR 2025 · similarity 0.89)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.89)
- BOND: Aligning LLMs with Best-of-N Distillation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.