Research map / Large language models / Language, safety and interpretability
MT-Ranker: Reference-free machine translation evaluation by inter-system ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes reference-free MT evaluation as pairwise ranking of two translations, training an mT5 encoder with NLI, human-vs-machine, and synthetic perturbation data, reaching state-of-the-art on DA20, MQM20/21/22, and ACES.
Problem
Machine translation evaluation is usually framed as absolute quality regression and mostly relies on references; the pairwise better-worse formulation for reference-free evaluation has been underexplored, and human score annotations are costly and inconsistent.
Approach
Model reference-free MT evaluation as a binary pairwise ranking over (source, translation A, translation B) using a multilingual T5 encoder with mean pooling and logistic regression, trained in three stages: cross-lingual NLI indirect supervision, human-vs-machine discrimination, and weakly supervised synthetic pairs (BERTScore-labeled DA translations and perturbed MT-PRISM references).
Key results
- On WMT20 DA, MT-Ranker-XXL averages 22.0 Kendall's Tau on X-to-En vs 17.6 for the best supervised baseline OpenKIWI-XLMR, and 52.2 on EN-to-X vs 48.4 for T5Score-XL_sup.
- On ACES, MT-Ranker-XXL achieves an ACES-Score of 18.46, above KG-BERTScore at 17.49.
- On MQM20/21/22, MT-Ranker-XXL reports averages of 22.3, 12.4 and 27.7, outperforming the nearest baselines by 1.3, 0.4 and 2.8 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reference-free pairwise ranking with MT-Ranker-XXL instead of OpenKIWI-XLMR | DA20 X-to-En evaluation · DA20 X-to-En Avg | Kendall's Tau: 22.0 vs 17.6 | Helped |
| Use reference-free pairwise ranking with MT-Ranker-XXL instead of supervised T5Score-XL | DA20 En-to-X evaluation · DA20 En-to-X Avg | Kendall's Tau: 52.2 vs 48.4 | Helped |
| Use MT-Ranker-XXL instead of COMET-QE | MQM20 evaluation · MQM20 | Kendall's Tau: 22.3 vs 21.0 | Helped |
| Use MT-Ranker-XXL instead of COMET-QE | MQM21 evaluation · MQM21 | Kendall's Tau: 12.4 vs 12.0 | Helped |
| Use MT-Ranker-XXL instead of CometKiwi | MQM22 evaluation · MQM22 | Kendall's Tau: 27.7 vs 25.9 | Helped |
| Use MT-Ranker-XXL instead of KG-BERTScore | ACES evaluation · ACES | ACES-Score: 18.46 vs 17.49 | Helped |
| Add cross-lingual NLI pretraining (Stage I) (ablation) | MT-Ranker-Large · DA20 | Kendall's Tau | Helped |
| Add human-versus-machine translation fine-tuning (Stage II) (ablation) | MT-Ranker-Large · DA20 | Kendall's Tau | Helped |
| Add weakly supervised synthetic-data fine-tuning (Stage III) (ablation) | MT-Ranker-Large · DA20 | Kendall's Tau | Helped |
| Fine-tune with human-annotated relative-ranking data | MT-Ranker · All benchmark datasets | Kendall's Tau | Helped |
Surprising findings
- Performance on the untranslated ACES phenomena is poor for untranslated-vs-ref-word and untranslated-vs-synonym, despite good copy-source results.
- Pairwise predictions are rarely intransitive, with inconsistencies under 1% of triples on the seven DA20 pairs where they occur.
Limitations
- Performs poorly on untranslated-error phenomena where the untranslated text resembles the source.
- Wrong-language errors are unsolvable for reference-free systems, and MT-Ranker-XXL scores -0.53 on that category.
- Relies on a few language pairs for MQM and a modified Kendall's Tau implementation that is not directly comparable to prior work.
Experimental setup
Datasets: XNLI WMT Direct Assessment 2017-2020 MT-PRISM WMT20 DA20 MQM20 MQM21 MQM22 ACES
Benchmarks: WMT20 DA20 MQM20 MQM21 MQM22 ACES
Metrics: segment-level Kendall's Tau ACES-Score system-level Pearson correlation
Baselines: sentBLEU BERTScore T5Score-XL COMET-QE OpenKIWI-XLMR UniTE CometKiwi KG-BERTScore
Builds on: COMET-QE BERTScore T5Score UniTE KG-BERTScore
Closest papers by meaning
- A Benchmark for Learning to Translate a New Language from One Grammar Book (ICLR 2024 · similarity 0.88)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.88)
- Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation (AAAI 2025 · similarity 0.87)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.87)
- Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost (NeurIPS 2025 · similarity 0.87)
- QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.