Research map / Label-efficient and robust learning / Recognition and long-tail
Towards Explainable Evaluation Metrics for Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Concept paper that defines explainability for MT evaluation metrics, surveys 37 works into a taxonomy of explanation types, and outlines future directions including LLM-generated natural language explanations.
Problem
Most high-quality MT evaluation metrics are black-box neural or LLM-based models, and their lack of transparency may be one reason classical metrics like BLEU remain dominant in practice.
Approach
Defines explainability goals, properties (faithfulness, plausibility, model access, explanation level and type, audiences) for MT metrics, and organizes prior explainable MT metric work into a taxonomy by explanation type (quality aspects, feature importance, fine-grained errors, perturbation robustness, linguistic properties).
Key results
- Literature search started from 11 seed papers, expanded to 383 citing papers, filtered to 177 by keyword, and yielded 12 papers on MT metric explainability; 37 papers are ultimately described.
- Proposes four explainability goals for MT metrics: diagnose and improve metrics, make metrics more accessible, support semi-automatic labeling, and check for social biases.
Surprising findings
- Explanations that are plausible to humans can still be unfaithful, and the paper notes that a faithful explanation cannot be established for neural metrics, making plausibility alone potentially misleading.
Limitations
- No experiments: the paper is a conceptual survey and does not empirically evaluate explanation methods.
- Faithfulness of current explanation methods is largely untested for MT metrics.
- Literature selection limited to neural-network-based metrics and a keyword-filtered citation tree.
Experimental setup
Builds on: BARTScore COMET BERTScore MQM
Closest papers by meaning
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.85)
- Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost (NeurIPS 2025 · similarity 0.84)
- Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.83)
- Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation (AAAI 2025 · similarity 0.82)
- Learning from others' mistakes: Finetuning machine translation models with span-level error annotations (ICML 2025 · similarity 0.79)
- Enhancing Machine Translation Experiences with Multilingual Knowledge Graphs (AAAI 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.