Research map / Large language models / Instruction and fine-tuning
Calibrating Translation Decoding with Quality Estimation on LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes calibrating LLM translation likelihoods to quality by directly optimizing a Pearson correlation loss (plus SFT regularizer) on sampled hypotheses, improving MAP/beam-search translation of Tower and ALMA models and yielding a likelihood that works as a QE signal.
Problem
MAP decoding in NMT and LLM translation is only weakly aligned with translation quality (beam search curse), so high-likelihood hypotheses are often poor and likelihood is a weak uncertainty/QE signal.
Approach
Train the LLM so that sequence log-likelihoods of multiple sampled hypotheses per source correlate (Pearson) with an external quality metric score, computed as a dot product of normalized values, combined with an SFT loss on the best hypothesis; this calibrates likelihood to quality and makes beam search more effective.
Key results
- Calibrated TowerInstruct-Mistral-7B improves average KIWI-XL by +2.8 and XCOMET by +2.7 over the baseline on WMT24 en-xx (off-policy).
- Calibrated TowerBase-13B with beam search is reported to be comparable to Tower-70B-v2 with 100-sample MBR/TRR while being about 200x faster (latency estimate).
- On ALMA-7B, calibration gives +0.5 COMET, +1.4 KIWI-XXL, +2.0 XCOMET over ALMA-7B-R on out-of-English WMT22; human eval en-zh mean 5.49/6 vs 5.0-range baseline with 106 wins vs 32 losses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Pearson likelihood-quality calibration instead of using TowerInstruct-Mistral-7B unchanged | TowerInstruct-Mistral-7B · WMT24 en→xx, 9 directions average | KIWI-XL: +2.8 points | Helped |
| Apply Pearson likelihood-quality calibration instead of using TowerInstruct-Mistral-7B unchanged | TowerInstruct-Mistral-7B · WMT24 en→xx, 9 directions average | XCOMET: +2.7 points | Helped |
| Use Pearson calibration rather than SFT on Best-of-N samples | TowerInstruct-Mistral-7B · WMT24, average across metrics | KIWI-XXL: +3.5 | Helped |
| Use Pearson calibration rather than SFT on Best-of-N samples | TowerInstruct-Mistral-7B · WMT24, average across metrics | XCOMET: +2.7 | Helped |
| Use Pearson calibration rather than SFT on Best-of-N samples | TowerInstruct-Mistral-7B · WMT24, average across metrics | COMET: +1.2 | Helped |
| Use COMET rather than KIWI-XXL as the calibration training signal (ablation) | TowerInstruct-Mistral-7B + Calibration · WMT24 | XCOMET: +1.4 | Helped |
| Increase the number of Best-of-N samples used for SFT (ablation) | Tower-7B and Tower-13B · WMT22 QE, en→de | Spearman correlation: from around 51.5 to 54.0 points | Helped |
| Scale the model from 7B to 13B (ablation) | Tower models · WMT22 QE | quality estimation ability | No clear effect |
| Increase the sampling size during calibration training (ablation) | TowerInstruct-Mistral-7B · WMT24 | KIWI-XL | Helped |
| Use beam search after calibration instead of Best-of-N sampling | TowerInstruct-Mistral-7B + Calibration · WMT24 en→de,es,ru,zh | COMET | Helped |
| Calibrate TowerInstruct-Mistral-7B | TowerInstruct-Mistral-7B · WMT24 en→zh human evaluation | human score: 137 vs. 67 for scores of 6; 106 vs. 32 | Helped |
Surprising findings
- SFT on best-of-N samples gives limited or even negative gains over strong translation-specialized baselines, while calibration gives consistent gains.
- COMET (reference-based) as the training signal is least effective, which the authors attribute to constraining hypotheses to resemble the reference.
Limitations
- Only open-source base models of limited language coverage were calibrated; languages beyond base model coverage untested.
- Exact MAP decoding and larger beam sizes were not explored.
- Relies on external metric models (CometKiwi-XXL) as reward signal, so metric gaming risk persists.
Experimental setup
Datasets: Flores-200 NTREX WMT22 WMT24 WMT22 QE (MQM)
Benchmarks: WMT24 WMT22
Metrics: CometKiwi-XL CometKiwi-XXL XCOMET COMET-22 Spearman Kendall human score 0-6
Baselines: TowerInstruct-7B TowerInstruct-13B TowerInstruct-Mistral-7B SFT on BoN data CPO DPO GPT-4o GPT-4o-mini Tower-70B-v2 + MBR/TRR ALMA-7B-R CometKiwi
Builds on: CPO Best-of-N sampling MBR decoding Tower ALMA
Closest papers by meaning
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.91)
- QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation (NeurIPS 2024 · similarity 0.91)
- MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods (ICLR 2024 · similarity 0.89)
- Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (ICML 2024 · similarity 0.88)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.87)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.