Research map / Large language models / Language, safety and interpretability
Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that averaging segment-level BLEU/chrF scores instead of corpus-level aggregation makes lexical metrics correlate much better with human judgements (WMT23 MQM/DA) and with bootstrap-robust scores.
Problem
Corpus-level aggregation of lexical MT metrics (BLEU, chrF) is the default, but it is statistically weak and may bias scores toward long sentences, reducing agreement with human judgements relative to neural metrics.
Approach
Frames corpus-level BLEU as a length-weighted average of sentence ratios (ratio of sums vs average of ratios) and shows that segment-level aggregation (averaging per-sentence scores) yields system-level scores that track bootstrap-robust scores and correlate much more strongly with human judgements, closer to neural metrics.
Key results
- On WMT23 MQM, m-chrF reaches Pearson 0.806 vs 0.392 for chrF, close to BERTScore (0.855) and BLEURT (0.872).
- m-BLEU correlates 0.776 with MQM human scores vs 0.425 for corpus-level BLEU.
- Downsampled corpus-level BLEU correlates with bootstrap-resampled scores at roughly the same level regardless of sample size, whereas segment-level scores increase in correlation as the number of samples grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace corpus-level BLEU with segment-level m-BLEU | BLEU system-level evaluation · WMT23 MQM | Pearson correlation: 0.425 -> 0.776 | Helped |
| Replace corpus-level chrF with segment-level m-chrF | chrF system-level evaluation · WMT23 MQM | Pearson correlation: 0.392 -> 0.806 | Helped |
| Replace corpus-level BLEU with segment-level m-BLEU | BLEU system-level evaluation · WMT23 DA | Pearson correlation: -0.006 -> 0.729 | Helped |
| Replace corpus-level chrF with segment-level m-chrF | chrF system-level evaluation · WMT23 DA | Pearson correlation: 0.285 -> 0.802 | Helped |
| Use segment-level aggregation instead of bootstrap-resampled aggregation | chrF system-level evaluation · WMT23 MQM | Pearson correlation: 0.804 -> 0.806 | Helped |
| Use segment-level aggregation instead of bootstrap-resampled aggregation | BLEU system-level evaluation · WMT23 MQM | Pearson correlation: 0.762 -> 0.776 | Helped |
| Use segment-level aggregation instead of bootstrap-resampled aggregation | chrF system-level evaluation · WMT23 DA | Pearson correlation: 0.793 -> 0.802 | Helped |
| Use segment-level aggregation instead of bootstrap-resampled aggregation | BLEU system-level evaluation · WMT23 DA | Pearson correlation: 0.456 -> 0.729 | Helped |
| Increase downsampled test-set size for segment-level aggregation | m-BLEU and m-chrF · WMT23 downsampled test sets | Correlation with bootstrap-resampled scores | Helped |
| Increase downsampled test-set size for corpus-level aggregation | BLEU and chrF · WMT23 downsampled test sets | Correlation with bootstrap-resampled scores | No clear effect |
Surprising findings
- Corpus-level BLEU scores with large test sets are about as correlated with bootstrap-robust scores as single-sample evaluations, suggesting corpus-level aggregation lacks statistical robustness.
- Under DA annotations, plain BLEU correlates at -0.006 with human judgements, while segment-level m-BLEU reaches 0.729.
Limitations
- Relies on a single data source (WMT23) and a single implementation (SacreBLEU) with default parameters.
- Only three language pairs with MQM and eight with DA are used for the human correlation analysis.
- Neural metric evaluated with a specific model variant (wmt22-cometkiwi-da) and only default settings.
Experimental setup
Datasets: WMT23 Metrics Shared Task dataset (492 system evaluations) WMT23 MQM annotations (three language pairs) WMT23 DA annotations (eight language pairs)
Benchmarks: WMT23 Metrics Shared Task MQM WMT23 Metrics Shared Task DA
Metrics: Pearson correlation boxplot distribution of downsampled correlations
Baselines: BLEU chrF COMET BLEURT BERTScore
Builds on: BLEU chrF SacreBLEU bootstrap resampling
Closest papers by meaning
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.87)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.86)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.84)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.84)
- Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.84)
- DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.