Research map / Large language models / Reasoning and chain of thought
Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically analyzes large reasoning models as MQM-style machine translation judges on WMT24, then proposes ThinMQM, fine-tuning LRMs on synthetic human-like error-span-then-score trajectories to cut thinking budget (~35x) and improve correlation.
Problem
Large reasoning models (LRMs) are underexplored as judges for machine translation quality under the MQM framework; they may overthink, overestimate scores, and respond inconsistently to evaluation materials (source/reference).
Approach
Analyze LRM-as-a-judge behaviour (evaluation materials, scoring mechanisms, thinking budgets) and calibrate LRM thinking by SFT on synthetic, human-like two-phase trajectories (error span annotation then rubric-based scoring) derived from WMT23 MQM data.
Key results
- ThinMQM raises R1-Distill-Qwen-7B average meta-evaluation score from 61.1 to 69.8 (+8.7).
- ThinMQM QwQ-32B reaches 72.2 average, versus 68.3 for base QwQ-32B and 71.9 for xCOMET.
- ThinMQM reduces thinking budget by roughly 35x; QwQ 32B inference time per 1,000 En-De examples drops from 12 minutes to 40 seconds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train QwQ 32B on synthetic human-like MQM evaluation trajectories | QwQ 32B · WMT24 Metrics (Avg.) | All: 68.3 -> 72.2 (+3.9) | Helped |
| Train R1-Distill-Llama-8B on synthetic human-like MQM evaluation trajectories | R1-Distill-Llama-8B · WMT24 Metrics (Avg.) | All: 64.9 -> 70.8 (+5.9) | Helped |
| Train R1-Distill-Qwen-7B on synthetic human-like MQM evaluation trajectories | R1-Distill-Qwen-7B · WMT24 Metrics (Avg.) | All: 61.1 -> 69.8 (+8.7) | Helped |
| Reduce the thinking budget using ThinMQM calibration | LRM evaluators · WMT24 Metrics | thinking budget: ~35x | Helped |
| Use the LRM itself for post-scoring instead of rule-based scoring (ablation) | QwQ 32B · WMT24 Metrics | mean performance: 68.3 -> 68.1 | Hurt |
| Remove ICL demonstrations from the QwQ 32B source-only setup (ablation) | QwQ 32B · WMT24 Metrics | All: 53.9 -> 52.8 | Hurt |
| Remove ICL demonstrations from the QwQ 32B joint setup (ablation) | QwQ 32B · WMT24 Metrics | All: 52.2 -> 51.5 | Hurt |
| Remove ICL demonstrations from the QwQ 32B reference-based setup (ablation) | QwQ 32B · WMT24 Metrics | SPA (%): 82.4 vs. 76.3 | Hurt |
| Remove ICL demonstrations from the QwQ 32B reference-based setup (ablation) | QwQ 32B · WMT24 Metrics | All: 51.5 vs. 51.5 | No clear effect |
| Replace the GEMBA prompt with paraphrased prompt P1 (ablation) | R1-Distill-Qwen-7B · WMT24 Metrics (Avg.) | All: 61.1 -> 62.5 | Helped |
| Replace the GEMBA prompt with paraphrased prompt P2 (ablation) | LRM baselines · WMT24 Metrics (Avg.) | All: 68.3 -> 68.1; 64.9 -> 62.6; 61.1 -> 59.7 | Hurt |
| Replace the GEMBA prompt with paraphrased prompts P1–P3 (ablation) | R1-Distill-Llama-8B and R1-Distill-Qwen-7B · WMT24 Metrics | Acc*eq: 68.0; 51.5 | No clear effect |
Surprising findings
- Source information is detrimental for small LRMs (7/8B) while reference information is harmful for larger LRMs, reversing the 'lost in source' pattern seen in LLMs.
- Thinking token budget is largely unrelated to model size or instance difficulty, and LRMs underperform their general-purpose LLM counterparts in nearly half of settings.
Limitations
- Synthetic thinking trajectories may not capture the diversity of human cognitive evaluation processes.
- Evaluation limited to WMT24 with only three language pairs, with risk of data contamination.
- Calibration targets efficiency and scoring distribution but not fine-grained error-type identification.
Experimental setup
Datasets: WMT23 MQM WMT24 Metrics MQM Hindi-Chinese expert MQM
Benchmarks: WMT24 Metrics MQM (En-De, En-Es, Ja-Zh) Hindi-Chinese expert MQM
Metrics: system-level SPA segment-level tie-calibrated pairwise accuracy system-level Pearson Kendall tau
Baselines: BLEU COMET-22 xCOMET GEMBA-ESA Gemini-2.5-Pro Deepseek-R1 QwQ 32B R1-Distill-Llama-8B R1-Distill-Qwen-7B
Builds on: GEMBA-MQM MQM framework DeepSeek-R1 COMET
Closest papers by meaning
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.90)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- Calibrating Reasoning in Language Models with Internal Consistency (NeurIPS 2024 · similarity 0.89)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.89)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.