Research map / Probabilistic and scientific modeling / Variational and generative sampling
Model-Based Minimum Bayes Risk Decoding for Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes model-based MBR (MBMBR), which replaces the Monte Carlo estimate of the MBR expected utility with the model's own (renormalized) probabilities over the unique sampled hypotheses, and shows it lowers KL divergence and often improves BLEU/ROUGE/CIDEr-style scores across MT, summarization, captioning and data-to-text.
Problem
Monte Carlo estimation in MBR decoding uses empirical counts of sampled hypotheses, which is a noisy estimate of the model distribution when the number of samples is small relative to the hypothesis space; the model probabilities of samples are usually available but ignored.
Approach
Use the model probability of each unique sampled hypothesis (renormalized over the sampled set) as the estimate of the distribution in the MBR objective, instead of sample frequency counts; this is shown to be the KL-information projection of the model onto distributions supported on the samples, plus an optional length normalization (MBMBR_L) that multiplies by exp(length) to counter length bias.
Key results
- On WMT'19 De-En with epsilon sampling (|H|=256), MBMBR_L reaches 35.86 BLEU vs 35.77 for MBR; MBMBR_L beats MBR in 47/56 settings across WMT'19 De-En/Ru-En and sampling methods.
- Model-based estimate reduces KL divergence from the model distribution; in a Zipf simulation the KL is 0.069 vs 0.183 for Monte Carlo, and about half as many samples are needed for equal divergence.
- On MS COCO with BLIP-2 and epsilon sampling, MBMBR outperforms MBR at all sample sizes (e.g., 34.96 vs 33.24 at 64 samples); however MBMBR drops on ancestral sampling and on E2E NLG with Mistral-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Monte Carlo probability estimates with model probabilities | MBR decoding · WMT'19 De-En and Ru-En, epsilon sampling | BLEU: #wins MBMBR{} > MBR{} is 43/56 | Helped |
| Replace Monte Carlo probability estimates with model probabilities | MBR decoding · IWSLT'17 Fr-En, XSum, SAMSum, CNN/DM, MS COCO, and nocaps | BLEU or ROUGE-L | Mixed |
| Add length normalization to model probabilities (ablation) | MBMBR · WMT'19 De-En and Ru-En | BLEU: #wins MBMBR\textsubscript{L}{} > MBR{} is 47/56 | Helped |
| Add length normalization to model probabilities (ablation) | MBMBR · XSum | ROUGE-L: 35.90 -> 36.18 | Helped |
| Add length normalization to model probabilities (ablation) | MBMBR · E2E NLG | BLEU | Mixed |
| Replace Monte Carlo probability estimates with model probabilities | MBR decoding · E2E NLG | BLEU: 22.30 -> 16.07 | Hurt |
| Use model probabilities instead of Monte Carlo probabilities | MBR distribution estimation · Controlled Zipf-distribution simulation | KL divergence: 0.183\;(\pm 0.054) -> 0.069\;(\pm 0.016) | Helped |
| Use model probabilities instead of Monte Carlo probabilities | MBR distribution estimation · WMT'19 De-En, IWSLT'17 Fr-En, XSum, and MS COCO | KL divergence | Helped |
| Use epsilon sampling instead of top-k, nucleus, or ancestral sampling (ablation) | MBR decoding · Text summarization and image captioning datasets | ROUGE-L or BLEU | Helped |
| Use model probabilities instead of Monte Carlo probabilities | MBR decoding with COMET-22 utility · WMT'19 De-En and Ru-En | COMET-22: 85.75 -> 85.84; 85.37 -> 85.44 | Helped |
| Use model probabilities instead of Monte Carlo probabilities | MBR decoding with BLEURT-20 utility · WMT'19 De-En | BLEURT-20: 28.31 -> 26.92 | Hurt |
| Use model probabilities instead of Monte Carlo probabilities | MBR decoding · IWSLT'17 Fr-En with TowerInstruct-7B-v0.1 | COMET-22: 85.47 -> 86.55 | Helped |
Surprising findings
- Plain MBMBR without length normalization often underperforms MBR on XSum and with top-k/nucleus sampling on MS COCO and nocaps, showing length bias from the raw model probability hurts.
- Using BLEURT-20 as the utility, MBMBR decreases performance relative to MBR on WMT'19 De-En, and with TowerInstruct/ALMA the gains do not consistently hold.
Limitations
- Gains are inconsistent across utilities, tasks and sampling methods; MBMBR underperforms MBR in several settings (e.g., ancestral sampling, E2E NLG, ALMA-7B-R).
- Length bias from raw model probabilities requires an ad hoc length normalization fix.
- Experiments use only the first 1000 inputs of each dataset due to compute constraints, and several results are single-run.
Experimental setup
Datasets: WMT'19 De-En WMT'19 Ru-En WMT'19 En-De WMT'19 En-Ru IWSLT'17 Fr-En XSum SAMSum CNN/DM MS COCO nocaps E2E NLG Zipf synthetic distribution
Benchmarks: WMT'19 IWSLT'17 Fr-En XSum SAMSum CNN/DM MS COCO nocaps E2E NLG
Metrics: BLEU ROUGE-L CIDEr COMET-22 BLEURT-20 KL divergence Jensen-Shannon divergence
Baselines: MBR (Monte Carlo estimate) ancestral sampling beam search with reranking
Builds on: minimum bayes risk decoding epsilon sampling BERTScore
Closest papers by meaning
- Uncertainty-Aware Decoding with Minimum Bayes Risk (ICLR 2025 · similarity 0.93)
- Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms (NeurIPS 2024 · similarity 0.90)
- Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategies (ICLR 2025 · similarity 0.88)
- MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods (ICLR 2024 · similarity 0.87)
- Reflection-Window Decoding: Text Generation with Selective Refinement (ICML 2025 · similarity 0.87)
- Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.