Research map / Label-efficient and robust learning / Recognition and long-tail
DirectQE: Direct Pretraining for Machine Translation Quality Estimation
Qu Cui, Shujian Huang, Jiahuan Li, Xiang Geng, Zaixiang Zheng, Guoping Huang, Jiajun Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DirectQE-Top-k instead of PLM-based QE | QE on WMT19 · WMT19 test | Pearson: 52.08 -> 55.08 | Improved |
| Use DirectQE-Top-k instead of PLM-based QE | QE on WMT19 · WMT19 test | F1-MULT: 37.04 -> 39.71 | Improved |
| Use DirectQE-Top-k instead of PLM-based QE | QE on WMT17 · WMT17 test | Pearson: 72.01 -> 73.56 | Improved |
| Use DirectQE ensemble instead of QE-Brain ensemble | Sentence-level QE · WMT17 test | Pearson: 71.59 -> 76.29 | Improved |
| Train the NMT-based QE system on generated pseudo pairs rather than parallel pairs (ablation) | NMT-based QE · WMT19 test | Pearson: 50.65 -> 51.50 | Improved |
| Train the NMT-based QE system on generated pseudo pairs rather than parallel pairs (ablation) | NMT-based QE · WMT17 test | Pearson: 67.44 -> 68.19 | Improved |
| Remove the sentence-level pretraining objective (ablation) | DirectQE-Top-k · WMT19 test | sentence-level QE: ($-$4.05) | Hurt |
| Remove the sentence-level pretraining objective (ablation) | DirectQE-Top-k · WMT19 test | word-level QE: ($-$2.61) | Hurt |
| Remove the sentence-level pretraining objective (ablation) | DirectQE-Top-k · WMT17 test | sentence-level QE: ($-$1.01) | Hurt |
| Increase pseudo-translation replacement rate from low to moderate (ablation) | DirectQE pseudo-data generation · Sentence-level QE | Pearson | Improved |
| Increase pseudo-translation replacement rate beyond the useful range (ablation) | DirectQE pseudo-data generation · Sentence-level QE | Pearson | Hurt |
| Use random replacement instead of quality-controlled sampling (ablation) | DirectQE pseudo-data generation · Sentence-level QE | Pearson | Hurt |
Later papers that compared with this method
1 later accepted paper reports DirectQE as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Denoising Pre-training for Machine Translation Quality Estimation with Curriculum Learning (AAAI 2023 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods (ICLR 2024 · similarity 0.87)
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.87)
- QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation (NeurIPS 2024 · similarity 0.86)
- Learning from others' mistakes: Finetuning machine translation models with span-level error annotations (ICML 2025 · similarity 0.86)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.85)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
