Research map / Large language models / Reasoning and chain of thought
Better Instruction-Following Through Minimum Bayes Risk
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that MBR decoding with LLM judges (mainly Prometheus-2-7B) improves instruction-following LLMs at test time, and that iterative DPO on MBR-selected outputs lets greedy decoding match MBR-decoded base models.
Problem
Reference-free LLM judges are weak for selecting outputs at test time, while reference-based judges are strong but no human references exist at inference. The paper asks whether MBR decoding with reference-based LLM judges can improve instruction-following LLMs, and whether the gains can be distille
Approach
Use candidate outputs as pseudo-references and a reference-based LLM judge (Prometheus-2-7B) as the MBR utility to pick the consensus candidate; then distill MBR-selected preference pairs into the model via iterative DPO so greedy decoding recovers MBR-level performance.
Key results
- Prometheus MBR decoding improves average AlpacaEval 2.0 win rate by +3.62 over greedy across five models (up to 70B), outperforming BoN (+1.74) and ROUGE/BERTScore MBR.
- MBR self-trained Llama-2-13b via three rounds of DPO (dpo-3-MBR) reaches 15.3 AlpacaEval 2.0 win rate with greedy decoding, above sft with MBR decoding (13.6).
- On MT-Bench, dpo-3-MBR 7B scores 6.14 vs 6.31-level MBR-decoded sft 7B at 5.78 and 13B sft greedy at 5.85, indicating self-trained greedy models exceed base models with MBR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Prometheus-2-7B reference-based MBR instead of greedy decoding | Llama-2 and Llama-3 instruction models · AlpacaEval 2.0 and MT-Bench | win rate (%) and score: +3.6% AlpacaEval 2.0; +0.28 MT-Bench | Helped |
| Use Prometheus reference-based MBR instead of Prometheus reference-free BoN | Llama-2 and Llama-3 instruction models · MT-Bench | score: 0.13 -> 0.28 Avg. Δ | Helped |
| Use ROUGE-1 or BERTScore as the MBR utility metric instead of greedy decoding | Llama-2 and Llama-3 instruction models · AlpacaEval 2.0 and MT-Bench | win rate (%) and score: +1.3% and +1.5% AlpacaEval 2.0; +0.11 and +0.08 MT-Bench | Helped |
| Use SFR-Embedder instead of greedy decoding as the MBR selection metric | Llama-2 and Llama-3 instruction models · AlpacaEval 2.0 and MT-Bench | win rate (%) and score: -1.4% AlpacaEval 2.0; +0.03 MT-Bench | Mixed |
| Use beam search instead of greedy decoding | Llama-2 and Llama-3 instruction models · AlpacaEval 2.0 and MT-Bench | win rate (%) and score: -0.5% AlpacaEval 2.0; -0.09 MT-Bench | Hurt |
| Select the longest candidate instead of greedy decoding | Llama-2 and Llama-3 instruction models · AlpacaEval 2.0 and MT-Bench | win rate (%) and score: -3.5% AlpacaEval 2.0; -0.04 MT-Bench | Hurt |
| Increase the MBR judge size from Prometheus-2-7B to Prometheus-2-8x7B | MBR decoding · MT-Bench | Avg. Δ score vs. greedy: 0.28 -> 0.39 | Helped |
| Use DPO self-training on MBR-selected preference pairs instead of SFT greedy | Llama-2-13B SFT · AlpacaEval 2.0 | LC win rate (%): 8.24 -> 15.3 | Helped |
| Use DPO self-training on MBR-selected preference pairs instead of SFT greedy | Llama-2-13B SFT · MT-Bench, first turn | score: 5.85 -> 6.75 | Helped |
| Use DPO rather than SFT for iterative MBR self-training | Llama-2-7B SFT · AlpacaEval 2.0 and MT-Bench | win rate (%) and score: DPO-3-MBR: 8.86 and 6.14; SFT-3-MBR: 6.48 and 5.51 | Helped |
| Use MBR-selected rather than BoN-selected preference pairs for DPO self-training | Llama-2-13B SFT, three training rounds · AlpacaEval 2.0 | LC win rate (%): 12.8 -> 15.3 | Helped |
| Increase DPO MBR self-training from round 2 to round 3 (ablation) | Llama-2-13B SFT · MT-Bench, first turn | score: 6.73 -> 6.75 | No clear effect |
Surprising findings
- SFR-Embedder MBR decoding reduces AlpacaEval performance relative to greedy decoding, while ROUGE and BERTScore MBR give small gains.
- SFT self-training on MBR-selected outputs yields much smaller gains than DPO self-training.
Limitations
- MBR decoding incurs substantial inference cost from N candidates and quadratic pairwise utility computation.
- Results depend on a generic single scoring rubric; the authors note rubric optimisation was not explored.
- Gains are smaller on reasoning-heavy categories such as math and coding.
Experimental setup
Datasets: UltraChat AlpacaEval MT-Bench
Benchmarks: AlpacaEval 2.0 MT-Bench
Metrics: length-controlled win rate MT-Bench score (1-10) Spearman rank correlation with GPT-4o
Baselines: greedy decoding beam search longest decoding best-of-N decoding ROUGE MBR BERTScore MBR SFR-Embedder MBR Universal Self-Consistency sft-full Starling-RM BoN
Builds on: Minimum Bayes Risk decoding Direct Preference Optimisation Prometheus-2 Universal Self-Consistency best-of-N decoding
Closest papers by meaning
- Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers (ICML 2024 · similarity 0.89)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.88)
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · similarity 0.88)
- NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.88)
- Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following (AAAI 2026 · similarity 0.88)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.