Research map / Large language models / Alignment and preferences
Fast Best-of-N Decoding via Speculative Rejection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Speculative Rejection, an inference-time alignment decoding method that starts Best-of-N with a very large batch and iteratively halts low-reward partial generations using a reward model, matching Best-of-N quality at 16-32x lower compute.
Problem
Best-of-N is a strong inference-time alignment method, but generating N full responses is computationally expensive and requires many GPUs for large N.
Approach
Begin with a very large batch that would OOM, then repeatedly score partial generations with a reward model and reject the lowest-scoring fraction alpha each time GPU memory nears its limit, so batch size shrinks dynamically while the best full response is still selected among survivors.
Key results
- With Llama-3-8B and RM-Mistral-7B, Speculative Rejection reaches a reward score that Best-of-N needs 16 to 32 GPUs to match, on a single GPU with similar latency.
- Win-rate vs Bo120 averaged across three models: Speculative Rejection (alpha=0.5) 66.17 WR and 70.01 LC-WR, versus 62.89 WR and 63.04 LC-WR for Bo3840 (ArmoRM reward).
- When maximizing average token log-probability, Speculative Rejection (alpha=0.5) achieves 39.9x average speedup over Bo3840 while obtaining lower perplexity (1.554 vs 1.698 average).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Speculative Rejection (α=0.5) instead of Bo3840 | AlpacaFarm-Eval generation; average over Mistral-7B, Llama-3-8B, and Llama-3-8B-Instruct · AlpacaFarm-Eval | WR: 62.89 -> 66.17 | Helped |
| Use Speculative Rejection (α=0.5) instead of Bo3840 | AlpacaFarm-Eval generation; average over Mistral-7B, Llama-3-8B, and Llama-3-8B-Instruct · AlpacaFarm-Eval | LC-WR: 63.04 -> 70.01 | Helped |
| Use Speculative Rejection (α=0.5) instead of Bo3840 | AlpacaFarm-Eval generation; average over Mistral-7B, Llama-3-8B, and Llama-3-8B-Instruct · AlpacaFarm-Eval | PPL: 1.698 -> 1.554 | Helped |
| Use Speculative Rejection (α=0.5) instead of Bo3840 | AlpacaFarm-Eval generation; average over Mistral-7B, Llama-3-8B, and Llama-3-8B-Instruct · AlpacaFarm-Eval | Speedup: 1.0× -> 39.9× | Helped |
Surprising findings
- On Llama-3-8B-Instruct, Speculative Rejection yields less improvement because the model generates shorter responses, resulting in fewer rejection rounds.
- Speculative Rejection with Mistral-7B yields a speedup partly attributed to inefficiency of HuggingFace GQA implementation, not only to the method.
Limitations
- Fixed rejection rate alpha across prompts; correlation between partial and final rewards varies by prompt.
- Relies on the statistical correlation between partial and final rewards; reward models are normally trained on full responses.
- No error bars reported due to compute cost; results averaged over 100 prompts.
Experimental setup
Datasets: AlpacaFarm-Eval
Benchmarks: AlpacaFarm-Eval
Metrics: relative GPU compute speedup improvement score win-rate length-controlled win-rate perplexity
Baselines: Best-of-N
Builds on: Best-of-N speculative decoding PagedAttention/vLLM
Closest papers by meaning
- Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment (ICML 2025 · similarity 0.91)
- GRIFFIN: Effective Token Alignment for Faster Speculative Decoding (NeurIPS 2025 · similarity 0.90)
- Approximately Aligned Decoding (NeurIPS 2025 · similarity 0.90)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.90)
- Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.90)
- Guided Speculative Inference for Efficient Test-Time Alignment of LLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.