Research map / Large language models / Language, safety and interpretability

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Specialist, a method that specializes a prompted LLM autorater to a fixed test set by using same-source historical human ratings as in-context examples, yielding large gains over XCOMET on WMT'23/'24 MQM machine translation evaluation.

Problem

Automatic evaluators are designed to generalize across test sets, but in practice evaluation uses a small fixed set of canonical test sets; the paper asks whether an LLM autorater can be specialized to a given test set using historical ratings.

Approach

For each test example, construct in-context demonstrations from historical human ratings of other systems' outputs on the same source input (pseudo-SxS, same rater), so the prompted LLM learns the test set's and rater's error behavior without any finetuning.

Key results

• Specialist AutoMQM improves character-level F1 by 54% over XCOMET-XXL-QE on WMT'23 (51.59 vs 33.50 averaged) and 119% on WMT'24 (35.59 vs 16.23).
• Only 3 ICL examples (from 3 systems) are sufficient for Specialist AutoMQM to outperform XCOMET on en-de and zh-en WMT'23.
• Parrot baseline (copying ICL errors) reaches only 27.6 F1 on en-de vs 45.7 for Specialist, indicating gains are not merely copying.

Measured effects reported in the paper

Surprising findings

• Specialist performance drops to shuffled-baseline level when ICL examples come from a different human rater, showing the method specializes to the rater as well as the test set.
• Specialist AutoMQM shows no gain on MT-Bench, where ratings were not collected pseudo-SxS and gains beyond inter-annotator agreement could not be measured.

Limitations

• Requires pseudo-SxS human ratings from multiple systems on the same test set, which is not standard practice and expensive to collect.
• Relies on proprietary LLM APIs; performance on highest-quality systems is weaker due to backbone translation limits.
• Results on MT-Bench inconclusive because ratings were not pseudo-SxS; en-es WMT'24 gains smaller due to non-pseudo-SxS rating.

Experimental setup

Datasets: WMT'23 MQM ratings (en-de, zh-en) WMT'24 MQM ratings (en-de, en-es, ja-zh) WMT'23 Round2/Round3 MQM ratings WMT'23 Multi-Rater Subset

Benchmarks: WMT'23 MQM test set WMT'24 MQM test set MT-Bench (limitation experiment)

Metrics: character-level F1 character-level precision character-level recall segment-level Acc23

Baselines: XCOMET-XXL-QE GEMBA-MQM-QE Shuffled sources Fixed, different source MetricX-24-QE

Builds on: GEMBA-MQM AutoMQM pseudo-SxS rating

Closest papers by meaning

• RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.89)
• Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.89)
• Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.89)
• Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (ICLR 2025 · similarity 0.88)
• Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.88)
• Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.88)