Research map / Large language models / Alignment and preferences
MetaMetrics: Calibrating Metrics for Generation Tasks Using Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaMetrics, a supervised meta-metric that calibrates combinations of existing metrics (via Gaussian process/Bayesian optimization or XGBoost with iterative pruning) to human preference scores across summarization, translation, QA, captioning, and reward-model scoring.
Problem
Existing evaluation metrics for generation tasks each capture only some aspects of human preference and are often poorly calibrated to human judgments across languages, domains, and modalities.
Approach
Learn linear weights (GP/Bayesian optimization) or a boosted-tree combiner (XGBoost with iterative feature pruning) over normalized scores from many existing metrics, optimizing a rank correlation (Kendall tau) or accuracy against human judgments, with reference-based and reference-free variants.
Key results
- On SummEval, XGBoost (Top 2) with LLM-based metrics reaches average Kendall 0.447 versus 0.362 for UniEval and 0.422 for G-Eval (GPT-4).
- On WMT23 MQM, MetaMetrics-MT XGBoost gets overall sys/seg avg-corr 0.825, matching XCOMET-Ensemble (0.825) and close to Weighted Avg (0.827).
- On RewardBench, MetaMetrics-RM GP scores 93.5 overall, state-of-the-art among listed models for GP, and outperforms the 70B Nemotron reward model on Reasoning and Chat categories in the paper's comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine top 2 selected metrics instead of using all metrics with GP (ablation) | MetaMetrics-SUM · SummEval | Kendall tau avg.: 0.433 -> 0.437 | Helped |
| Use top 2 selected metrics instead of all metrics with XGBoost (ablation) | MetaMetrics-SUM · SummEval | Kendall tau avg.: 0.424 -> 0.447 | Helped |
| Replace linear metric weighting with multiplicative pairwise scoring (ablation) | MetaMetrics GP · Generation tasks | human-preference alignment | Hurt |
| Use iterative feature pruning and retain the top 5 metrics (ablation) | MetaMetrics-QA XGBoost · Combined QA | Kendall tau: 0.536 -> 0.528 | Hurt |
| Use iterative feature pruning and select the best iteration (ablation) | MetaMetrics-QA XGBoost · Combined QA | Kendall tau: 0.536 -> 0.538 | Helped |
| Use only the top 5 metrics instead of all metrics with GP (ablation) | MetaMetrics-QA GP · Combined QA | Kendall tau: 0.512 -> 0.512 | No clear effect |
| Use iterative feature pruning and retain top 5 metrics (ablation) | MetaMetrics-SUM XGBoost · BenchmarkLLM faithfulness | Kendall tau: 0.406 -> 0.377 | Hurt |
Surprising findings
- Linear weighting outperformed the multiplicative pairwise-product GP alternative.
- The paper reports the RewardBench-based 8B-scale metric beating 70B-parameter reward models in overall score claims, attributed to using fewer parameters.
Limitations
- Excludes LLM-based metrics above 10.7B parameters due to compute constraints.
- Does not explore exhaustive combinations of LLM-based metrics.
- RLHF with PPO left for future work.
Experimental setup
Datasets: SummEval BenchmarkLLM CNN/DailyMail XSUM WMT20-22 MQM WMT23 MQM WMT24 EQAE EVOUNA Natural Questions TriviaQA NarrativeQA
Benchmarks: SummEval BenchmarkLLM WMT23 MQM EQAE/EVOUNA QA Flickr8k-Expert THumB 1.0 RewardBench
Metrics: Kendall tau Spearman correlation Pearson correlation accuracy (acc-t) RewardBench accuracy
Baselines: BLEU chrF METEOR BERTScore BLEURT UniEval G-Eval MetricX-23-XXL XCOMET-Ensemble Llama-3.1-Nemotron-70B-Reward Skywork-Reward-Gemma-2-27B Uniform ensemble
Builds on: XGBoost Gaussian process Bayesian optimization COMET UniEval G-Eval
Closest papers by meaning
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.86)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.86)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.85)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.85)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.85)
- Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.